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. CftAPTER 1 ' ■ * . ' 

• . . .* INTRODUCTION • 

This manual is, designed 'for district and school personnel who 
have the responsibility for conducting ah assessment using NAEP's 

EDUCATION FOR CITIZENSHIP: • A BICENTENNIAL SURVEY. 

• ■ ■< . <■ 

Since this assessment kit is being- made available to over 1600 
districts and schools it is difficult to define a uniform set of 
procedures that would apply to everyone. Some districts and 
schools have been conducting assessments for a number of years. 
♦For others, this may be n first assessment effort. Some districts 
have testing and evaluation staffs' already trained to carry out 
an assessment. For others, conducting this assessment will re- 
quire that a considerable amount of additional time and effort 

be pi aced^on van already overloaded staff. We tried to keep all 

.-. , ' * ' < 

of these factors 1n mind as 'this manual was developed and we hope 

a ■ r ' • ■ " .• . 

it has something to offer to each of you. ^ 

The chapters, i a this manual are orgar a in the order in which 
you will* be conducting your assessment- We suggest that you 
first review the entire manual from cover to cover. Then, re- 
examine in more depth those chapters that raise Questions in 
your mind. Depending upon the number <fr* people assigned to 
your assessment 6f fort, you may want to reproduce extra copies of 
certain chapters or perhaps the entire manual . 



The primary focus of the manual is to highlight the requirements 
for replicating the NAEP model for assessment, 'If you plan to 
conduct your assessment so that comparisons of your students 1 



performance levels can validly be made to the, NAEP results, it is 
very important that you follow these requirements. When you make 
your final data comparisons and discover differences between your 
students and the national sample, you wi 4 ll. want to feel confident 
that these differences are "real", and not a result of using 
different assessment procedures. 

'As stated /fn the Q & A brochures that are included in 'the assess- 
ment kit, NAEP will not.be able to provide additional assistance 
in helping you organize and conduct your assessment. While we ^ 
would 1 ijce to provide a more personalized .service, our staffing 



limitations will not allow us to do s'o. 



/ 



In a few months, we will ,be sending a questionnaire to solicit 
your comments on the utility of this citizenship assessment kit. 
We hope you Jot down any problems you encounter ,in y|^9 the ' 
assessment booklet, cassette tape-or this manual: so that when the 
questionnaire arrives, you will be able to give us a candid 
evaluation of the kit's strengths 'and weaknesses. 



CHAPTER 2 

BECOMING FAMILIAR WITH THE ASSESSMENT BOOKLET 

The booklet in your assessment 'kit is identical to the ones 
.used by NAEP in* the 197&/6 survey. You may have already noticed 
several design and layout features tha,t make the booklet appear 
quite (Afferent from' other tests you have administered. Some of 
these features will be important to your assessment activities 
an&.will be covered in detail- in different chapters'of this manual, 

« ' ' : f 

Other features are unique to NAEP* s printing and scoring require- 

ment^and you will not have to be .concerned with them. Take a 

few moments to become more famil/ar with the booklet by following 

1 / / " f " 
the 'steps bel ow: /' / ' ' " 

'.'* • - : • . / I * • ' * 

• First, look careful ly/at the cover page. If you requested 

/ / * ^ ^ 

an assessment kit. for 13-yearf-olds, you "should have a book- 
• ; .'•■../• ' I 
. let labeled "Package No. 9, fear 07 , Age Class 2 ." If 

*- * v . j *! } 

# you requested a kit for 17-yjear-olds, the booklet should" - 
* y be labeled "Package No. 9, Year 07, Age Class 3 ." While ' 

most of the questions in both booklets are identical , there 
are a couple of items in the 17-year-old booklet that are 
not in the 13-year-old booklet. These differences will be 
. discussed later. / * 

• At the bottom of the cover page are several sets of boxes* * 
andwals that NAEP used for recording*the students' grade, 

, sex , birthrate, race, etc. You also may find it convenient 



-to -code fnformatTon for^-yoV stents iri'.tbi-s section. The 
coding of this 1nforma^n r jjfll ie covered in Jater chapters 
of this manual . / * /■■ • - - ' -fa ' /' . 



On pages 2 and 3 ari?ex/fnp'l e questions Texercises) that the 
students should comple 

ment questions. The purpose of /these example questions -is . 

/ • / 

to acquaint the students with both multiple-choice and open- 
ended formats* to stress that only one'answer oval be filled 
in, for each question and to introduce them to the use of the 
"I don't know" response choice. - 

► The survey questions begin on page 4. All questions in the 
booklet are multiple-choice except question number 12 on 

-page 15. This question will involve a more extensive scor- 
ing effort and will be discussed later. 



> /■ 
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Notice that only one question is printedjier page. NAEP 
has followed this formal in all previous assessments so , 
that students will concentrate on one question at a time. 
Since results are reported for each. question it is impor- \ 

tant that students be- given an ample opportunity to respond 

■/> ■ ; ■ ' ■•• •••• •'• ' - -'X ' . 

withput being influenced by seeing' easier or harder questions 

7 tha4may be coming up. The . stop/signs' in "the lower right- o 

hand, corner of each page,, plus the instruction, "DO NOT 

CONTINUE UNTIL TOLD TO DO SO", are intended to discourage 

the students from working ahead in the booklet. 



On- pages 38 and 39 of the 13-year-old booklet (Age Class 2) 

'^you will see information labeled "Sources" and "Exercise 
■ " ' . - ■ . V , • 

Documentation".. This was for NAEPis use- only and you win 

not need to be«concerned-with these two pages. > 



• The back page. of the booklet contains several questions 
• concerning the home environment of the student', eg. , re- 
ceiving newspapers and magazines < number of books and en- 
cyclopedias in the home, father's and mother's education, 
and place of residence" on student's 9th or 13th birthday. 
^ NAEP will provide information on" the national sample of • 
13- and 17-year-olds for questions 5 and* 6 only : "How 
much school did your (father) (mother) complete?" You 
may want to ask your students the other* questions as well, 

but keep in mind that no. NAEP data will be available; 

» . . .. * % • 



• The booklets .were designed to be machine-scored J>y NAEP 
V ■ ' 

for the 1975-76 survey. No separate answer Sheets were 

used. The black lines that'appear along^fehe stapled- edge 
of each page were for NAEP *s scoring specifications and', 
you need not be concerned with them. The chapter* on r 
scaring wil. J suggest how you might proceed with your scor- 
ing task. 



•/You may also have. noticed the oval sets that appear in 
/ the lower left-hand corner of each page. These were 
for NAEP's use pnly. 



We recommend that you block out the NAEP logo on <he cover page be 
fore reproducing" the booklets. This will minimise possible mis- 
understandings that your assessment efforts are under the auspices 

• • • 

'of National Assessment. NAEP did receive clearance from the Office 
of Management and Budget (OMfi) to administer, the booklets nation- 
wide and the expiration' date, as noted on the cover, is September 
30, 1976. This expiration date does not apply' to your use of -the 

booklets. You may-- want to redesign the entire cover page so that 
... . * . ■ K. .'• 

it is readily identified as your citizenship' survey. 
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BECOMING FAMILIAR WITH THE CASSETTE TAPE \ " 

The cassette tape is designed to be used in conjunction with .the 

booklets during the assessment. Whether -you requested a. survey lei t 

for 13-year-blds or 17-year-olds or both, your kit should contain- 

only one tape. On* side of the cassette tape 1s labeled "PACKAGE 

V N0. 9, AGE CLASS 2," 13-YEAR-0LD$". and the other side is labeled 

"PACKAGE NO. 9. AGE . CLASS 3, 17-YEAR-OLDS". It is' important that 
• . • ' ' ' • t 

'you use the correct side of the cassetfe tape for.thejarticular 

.age-level assessment you will, be doing. 

t ' * • * . ""* ' .' / 

This tape recording procedure is used by NA£P for nearly all of 

its assessments. A professional, announcer reads each question . 

to the students as they. follow ?J'png 1n the booklet. After each* 

question 1s read, a slteVit pause appears on the ^oe arid allows' 

. the students time to mark their answers. Once the tape i smarted, 

the announcer provides -instruct tons to the students! beginning 6n 

• -•<..• 
page 2 of the booklet and leads them through the entire booklet. 

'The questions on the lust page of the- booklet concerning the number 
of magazines or books 1n the home, the level of parental education, 
etc., ar^_not presented on the tape since .the test administrator - 
may need to^1axi# or help the students, select the most appropriate 
response. There are several advantages to using the tape for your • 
Assessment: 1 
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• By using the tape you will know that yoiir students were 
; assessed under the 5 same conditions as the NAEP assessment 

conditions. This is important in order jto make accurate 

- < ' ■ ^ 11 • 



data. comparisons.' Theytape will also help to ensure 
standardization shoyla yofi administer the survey in more 
than .one setting. / y 

• Since \the tape paces the presentation of each. quest ion • 
you have control over the tojal length of the assessment 

1 period.. All students begin and end at the same time. 
Students who "have a tendency to rush through a test /in 

order to leave the testing site early are discouraged 

» * ■ 

from doing so. Therefore, distractions during the assess- 

■ . 

, ment period are held to a minimum. 

• The announcer serves as a test administrated" heard but 
not seen". Since no student questions are permitted once 
tf^e tape presentation 'begins, your test administrator is 

-< completely ffee to observe and monitor. 

• The use of the tape recording in conjunction with the booi 

• let m6st likely will be a unique experijnc£-f of your stu- 
dents. NAEP' receives tfiany conwerrtsfrom students wfio say, 
"Gee, this was f un. ^AX really didn't' seem like a test at 

-'all'." 



V 



During a testing situation it is desirable to have the 
students do their best. Since citizenship is being assess- 
ed, you want to collect information on their knowledge of 
the citizenship-questions. Poor readers may'be at a dis- 
advantage 1 in responding correctly, not because of their 
lack of knowledge concerning the citizenship questions 

12 
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tions due to t&eir .redding handicap. The s tape may, there- 
fore Jj proy We these students wfth a better understanding 
.. of each citizenship Question. ' x : 

* j ■ *• ■ -m. 

. As. a next stjep;, listen to the cassette tape. and' fo.l low along in the 
• booklet.' Thfe tape begins with the announcer saying "NOW OPEN YOUR 
BOOKLETS TO' PAGE I. After -reading the* example exercises, he 
proceeds to direct you through the booklet. * » 

IMPORTANT: Before using your tape for an assessment, make sure 
"the cassette is properly lab&ed. This can readily be checked by 
,. listening to the- announcer -as he reads the directions on page 2. 
The example question (exercise) for 1 3-yea.r-ol ds is "How many days 
*re there in one week?" The example question for 17-yearVolds is 
"The Mississippi is aifejo^TverQocated in which one of the fol- 
lowing countries?" 

When all of the questions have been completed, the announcer^says^ 
"STOP. NOW CLOSE YOUR BOOKLET AND LISTEN CAREFULLY FOR FURTHER 
DIRECTIONS." At this point" the te|t administrator would as^ the 
students to complete the- questions on the back page. ' - 

It should be pointed out* that your tapes do not have an introduction 
for the students stating what the survey; is about and why they are 
being assessed.. The standard NAEP introduction is not included on 
your tapes because^ it would -not be totally appropriate for your 
survey situation. You may want to prepare .some "introductory comments 
that the test administrator could read to the students prior to 
.beginning the tape presentation. • - 



The total tape time for the 13-year-olds 1 presentation is ap'proxi 
mately 36 minutes. The 17-year-olds' tape presentation runs^for 
30 minutes. 



5 — 
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CHAPTER- 4 

OVERVIEW OF REQUIREMENTS /OR REPLICATING THE^EP SURVEY 



Should you decide to repHcate NAEP's procedures in order to make 
valid comparisons between your-student^and those students in the 
national survey, there are certa ^requirements that you will need 
to follow. In summary, a NAEP replication fluring the 1976-77 *■ 
school year will require you to: j 

Assess your. 13-yeacrolds sometime during October-December 

i97s: • . . <■ . ■ % 

• Assess your 17-year-olds sometime during March-April 1977. 

• • . . . ■ s , 

• Reproduce i^e baolclets in ample quantity so that each 
student has his/her own booklet and answers each question, 
directly in the booklet, 

• Use the cassette tape for the assessment* (Reproduce 

T 

additional tapes if necessary. ) ; " 

• Score the questions according to NAEP's scoring keys and 
guides. / ■ 

• * • . .■ * 

Each requirement is discussed in more detail in chapters that fal- 
low. Any deviations you make from the required procedures could pos 
sibly destroy your ability to make valid comparisons to the NAEP 
data.. ..Therefore, the procedures .should be followed carefully . v 



7 *. 
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CHAPTER 5 



SETTING YOUR ASSESSMENT SCHEDULE 

If" you plan to conduct your assessment during the 1976-77 school 
^ear,-the 13-year-?old students should be assessed sometime during 
October-December 1976. This assessment period will assure that 
your 13-year-old.s were surveyed during the same time of the school - 
•year as the nationaKsampl^. Conducting your assessment a't a later 
v time could jeopardize tHe data comparisons because of your students 1 
exposure to additional schooling. A mid-November assessment period 
would 'be ideal. ■ i 

« 

The 17-year-ol-d students should be assessed sometime during March.- 
• April 1977. 



C* > . * • . ....*. 
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CHAPTER 6 

• • ■ . > ■ i% - 

SELECTING STUDENTS TO-BE ASSESSED 

Eligibility of students assessed by NASP is determined -by age- 
criteHa, not grade-criteria. Thus, 13- and 17-year-olds are 
. selected for t assessment without regard to t;ieir grade-level. If - 
you plan to assess your 13-year-olds this fall or your 17-year- 
olds next spring, the NAEP age-eligibility definitions are! 

r — ■* — — 1 - . 

13-year-olds: Any student born January 1 , 1963- 
December 3'i , 1 963 

17-year-olds: Any student bom October 1, 1959- 
September 30, 1960 

There are three exceptions tc the above age-eligibility defini- 
tions: . 

■9 

• educable mentally retarded. (EMR) students 

• functionally disabled students 

• non-English speaking students r 

NAEJ^does not assess any students that fall into any one of these cate- 
gories so they should not be assessed by you. NAEP doeV not provide 
.detailed criteria for determining whether a student should net be 
assessed 'but rather relies on school personnel to exclude such ' 
students. 

For this special citizenship- assessment survey, NAEP will be report- 
ing both age-level and grade-level results in an attempt to provide 
national student data that is maximally useful to districts and '* 
schools. Tnis means that age data will be reported for 13- and 



V- • / 

17-year-old students regardless of grade-lev.el . In addition, 
data will be reported for 13-year-olds who were in the 8th grade" 
and for 17 -year- olds, who were in, the 11th grade. While this data 
does not completely sample the 8th or 11th grade students (only 
13- er 17-year-olds^re included)- it does open the door for you to 
6t*idus$ a grade-level assessment by following special procedures. 
Pi itajor requirement of these procedures is foryou to identify your' 
8th and/or 11th grade students by age 'so that the survey results 
"for your 13- and 17-year-olds within these grades caff be .isolated 
and Clouted separately during analyses of your data. 

In surarcary, you have the option of setec£frig students according to 
an age or grade definition. Both plans have advantages find' di sad-' 
vantages and you will need to decide whjch.plan is most appropriate 
in light of your particular situation and needs. Assessing Strictly 
by age (13 and/or 17) will give you a trueJIAEP replication and -will 
permit you to compare your students directly to the national sample. 
However, this procedure may prevent assessing students within their 
classrooms -because the selected students' will becoming from several 
different grades-levels. , 

Assessing by grade (8th and/or 11th) also will allow you to make 
valid data comparisons, provided you identify your students by age 
as wep, so that results can be tabulated for only the 13- and/or 
17-year-olds. By using this procedure you also will have data for 
your entire 8th and/or 11th grade. Assessing by grade does allow 
you to administer the survey within intact classrooms. 



If the number of students enrolled in your district or school is 7 
not too larje to manage from an assessing and scoring standpoint, " 
you may decide to assess all age or grade eligible students and 
therefore, »you will not need to design a plan for sampling. School 
systems with large enrollments, however, may want to select a sample 
of students in order to reduce the_efforts and costs involved with 
printing, assessing and scoring the booklets. Keep in mind that 
"sampling is not a requirement for replicating the NAEP survey. A 
discussion on sampling procedures is included in Appendix A. The 
discussion provides a "ruleWf^humb" to fpllow in determining 
.whether you should assess. all eligible students or only a sample 
and also describes various sampling procedures to follow in'select- 
ing students> . 

Whether or not sampling is used, it wilV be necessary to prepare 
a list of students 7 to be assessed. This student roster, arranged 
in alphabetical order by- student name, will assist the test admin- 
istrator in distributing the booklets as the students arrive for 
assessment. An example NAEP roster for 13-year-old students is 
shown below: 

(3). (4) (5) . (6) 



1.0. 



Student 



• 


Adams, Sue 


8 


F 


Aug.. 2, 1963 


B 




Baker, John 


• 8 


M 


Feb. 14, 1963 


W 


> 


Carson, Betty 


7 


F 


Oct., 9, 1963 


w 
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When NAEP prepares student rosters for its assessments* the iden* 
^ification number (I. b.) column on the left 1s blank until the 
day of assessing. Serial I.D.s are written on/ the cover page 
Qf the booklets and as each student is given a booklet, the test 
^administrator records the 1.0. numbers on the roster- If your \ 
school already has'a student numbering system, you may want $o 
? use the student numbers as I*D. numbers and record themjw^ — -~ 
roster first. Then, when the'booklets^re -distributed, the test 
administrator should transcrl^e 7 ^^ I.D. number : that appears on 
the roster to the cover page of the student's booklet. * 

The student roster will serve as your master control sheet for 

later transcription of the student's grade (column 3) , sex (column 

' • • ' ' /\ 
4), birthdate (column 5) and race (column 6) to the booklet covers. 

- '■ ■ ' * . 

It 1s therefore very important that the I.D.s on the booklet 0 

cover page match the I.D.s- shoWn on the roster. 

"V', ■ 

0i ■ . .: ■ 
NAEP uses the I.D. mjmber system for its assessments because stii- 

dents do not write%feir names on the booklets ancf the rosters 

never leave the school building. "You may be able to simplify the 

procedures discussed above, if there is no concern with students 

/ placing their names directly on the booklets. 
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CHAPTER 
PRErASsfsSMENT 

Prior to the dayjs) of assessing, there are several tasks that will 
need to < be- performed. These are 1 isted in the check! 1st below. 
Some tasks, may not, be applicable to your particular 'assessment 
situation and you\may need to add others. 

' ' • . • V . . * ' 

* Check out tapVrecorder and cassette tape for malfunctions. 
D Sharpen pencils. 

Reserve room or auditorium for assessing. (Are there tables 

or arm chairs for students to' use in writing?) 

* * 

□ If an auditorium Is used for assessing, 1s tape recorder 
amplification adequate? 

□ -Pfepare a roster of students to be assessed! Show student's 
name, grade, sex, birthdate and race. This roster will be 
used -to check in students. as they arrive. 

□ If necessary, prepare and distribute parental pem^ssibn 
letters for students being assessed. / 

□ Reproduce booklets in sufficient quantities so each student 
will have a booklet. 

□ Enter I.D. numbers in the I.D. boxes on the front coyer page 
of- each booklet. For example, if you are assessing 200 



students, enter codes 00001 thru 00200. These I.O.s will 
later be recorded oiythe roster when the students arrive for 
assessment. (NOTE: This task will not be necessary ,1f you 
plan to use a roster that already has I.D. numbers assigned.) 

□ Notify students selected as to time and place of assessment. 

□ If neces^aryr-<iistribute Student Permit Passes for leaving/ 
entering classes. 

□ Prepare an introduction for test administrator to read to 
students before assessing begins. " • .. 

□ . Other : ■ g ' • 

□ Other: __ 

A * 

□ Other: __ 1 . 7 
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: CHAPTJR 8 1 
- .. * • • ' ASSESSMENT- TASKS 

Prior to the students' arrival at the assessment site, s check to be 
sure the correct side of the tape is placed in the recorder. As 
the students arrive; check them off on"the roster, and give each 
student a booklet and two. pencils. Enter on \he roster the I<D; 
number that you pre-recorded on the booklet cover or, if the roster 

■ * 

was pre-numbered, -write the I.D, number on the booklet cover page, 
Instruct each student not to open the booklet untlttold to do. so. 

* to* 

Since the tape does not begin with a general introduction that de- 
ar *- 

scribes the assessment, why the survey is being given, etc., the 
test : administrator or' other staff person may want to make some 
introductory comments at this /time. -If you are doing an age-level 
assessment following NAEP's schedule for assessment and' NAEP's age 
definitions, be aware that you most 'likely have selected some 
students who have not quite reached their 13th or. 17th' birthday: 
For this reason, it will cause less confusion during the assess- 
ment if no general, announcement is made that the survey fs being- 
given to-only. 13- and/or 17-year-olds,.' The test.administratpr 
could talk in terms of, "STUDENTS *Y0UR 'AGE ARE "BEING ASSESSED TO. .." 

After the introductory comments, the. test administrator should say, 
"I*AM GOING TO TURN ON A TAPE RECORDER AND YOU WILL HEAR ON THE TAPE 
WHAT WE WANT YOU TO DO. LISTEN TO THE TAPE CAREFULLY. "oNCE THE 
RECORDING STARTS, I CANNOT ANSWER ANY QUESTIONS. IS EVERYONE READY? 
IF SO, I WILL TURN ON THE TAPE RECORDER." 



•3 . 



Turn on the 'tape recorder. Be sure the tape is audible to all 

students. As the students complete the example exercise on page 

2 of the booklet, check to see that they, are marking their answers 

*'•'•• - \ 

properly. • Give assistances necessary. ' ' 



The tape presentation will run for approximately 30 minutes. When 
the- announcer says, "STOP. NOW CLOSE YOUR BOOKLET AND LISTEN CARE- 
' FULLY FOR FURTHER DIRECTIONS" , .turn off" the tape recorder. 

••/.:'■' .* 

'I - -.'- ■* ...... 

*: ' ». ■- • 1 * 

The test administrator should now direct the students to the ques- 

+ . ■ •••••• ' iK . ■ '"• ; .• • . 

tions on the back cover of the booklet. Read alj questions and 

i • '. . ., '' ■•.•'.* 

. response choices aloud to the. students andfobserve to see, .that they 

are filling in only, one oval. Give any assistance necessary to. 
obtain the best possible information. / Keep in mind that N AEP will- 
be providing data for. questions 5 and 6 only . If the students 
have any difficulty in understanding these two questions concern- 
ing' the highest* level 'of education completed by. their father and 
mother, the following definitions may be helpful-. 

• If the student' answers , "I don't know" to question 5 
or 6 you may be able to determine the correct answer by 
' - asking for the occupation of the pare/it and additional 

questions concerning schools the parent may have attended, 
"such as .nursing school, etc. However,' if you are not • 
^ able to determine .the correct educational category from 
the student's answers* the "I don't know" oval should be 
filled in. 



y 
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• In theabsence-of natural or adoptive parents, questions 

»■".■'> * • 

5 and 6 apply to the male and/or female In *the student's ' - 
home who 1s responsible for the day-by-day care of the 
student; that is, a step-parent, a grandparent, some other 
adult relative, or an unrelated person, such as a foster 
parent* If either the male or female parent-figure is 
absent from the home, the student should still answer to 
the best of his/her knowledge about the education of 
that parent-figure. — — — : — - 

^Since NAEP will be reporting results for questions 5 and 6 only, • 

you may want to redesign the back page. and add your own set of 

questions. • \ , ' \'. 

>'■/■'. • ' 

The total time required' to checkp/n the students, distribute the 

materials and introduce and administer the survey, should takeliO 

more than one hour. 
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CHAPTER 9. 

/ POST-ASSESSMENT TASKS' ." V 

When the asltssment is completed you will seed to transcribe infor- 

matfon from the student roster to the coding boxes on the cover 

...... •■ /. • * 

page^of each booklets The procedure described below will identify 
each booklet. according -to the student's grade, sex, bir£hdate and 
race. It is important that this information- be transcribed ac- - 
curately if you plan to analyze your results for these categories. 



% Booklet . 
Coding Boxes 

'• 3 

4 . 



Information 

Grade 

Sex 

Birthdate 
Race 



Code . 
Two Digits 
One Digit 



Four Digits 
. (month and last 
two digits of 
year) 



Sample 

. 8th grade = 08 



Male ' 
Female 



1 
2 



May 1963 \ =0563 



One Digit 



White 
Black 
Spanish 
• Heritage 
Other 



1 
2 

3 
4 



7 
8 



For Your Use 
For* Your Use 



Two Digits 
Two Digits 



Coding boxes 7 and 8 are available-for your use to add any other 
variables you feel wouTd-be important. For example, you may want - 
to identify each student by. school or by class. These coding boxes 
could be use$ for these variables. 



NOTE: NAE0 determines racial identification by visual observation 
arid use of surname as the students arrive at the assessment 
site. Although information is recorded on the student ros- 
..• ter'and later transcribed to the booklet coding boxes for 
the four race categories, data is reported for white anjl 

black only. • : i^-'" 

/ ■ 
Shown below is the sample 13-year-old roster that was presented in 

Chapter 6. » ' 



I.0#* 



Student 



C3) (4) 



(5) 



C6) 



numuer 


name 

Adams, Sue 


8 


F 


Aug. 2, 1963 


B 




Baker, John 


8 ' 


M 


Feb. 14, 1963 


W 


/* 

/ 


Carson, Betty 

j j . 


7 " 


F 


Oct. 9, 1963 


w 















in this example, the roster was prepared prior to the day of assess- 
ing. I. D. numbers had been serially written on the booklet covers. 
Since John Baker came the assessment site first, he was given 
the first booklet-with I fD.. number 00001. The test administrator" 
nvHoH thic number next to John's name on the roster. 
Betty Carson was given the booklet with I.D. number 00002 and Sue 
taams received booklet 00003. After the v assessment was completed 
ind all booklets handed in, the test administrator tra7rscHbe.d^the 
remaining information (grade, sex, birthdate and race) from the 
roster to the coding' boxes. 
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As a final example, the booklet .cover for Sue Adams looked .like 



this: 



5151 g 



CDCD 



CDCD 
CD CD 
CD CD 
CD CD 
CDCD 
CD CD 
CD CD 
CD CD 



CD 



0 




6 


3 



GDGDCD 
CDCDCD 
CDC±3CD 
CDCDCD 
CDCDCD 
CDCDCD 
CDCDCD 
CDCDCD 



6 


7 


8 




Ml- - 




m< 3 •> 


CDCD 


CD 


* ■ t T \eri 


CDCD 


CC" 1 


CD CD 


GDCO 


CD 


CDCD 


CDCD 


CD 


CD CD 


CDCD 




CDCD 


CDCD 




. CDCD 


,CDCD 




CDCD 


CDCD 




CDCD 


CDCD 




CDCD 


CDCD 



CDCDCD CD CD 



I. D. 
NUMBER 

0\Q\dK>i3 



CD CD CD CD CD 
CD CD CD CD CD 
CDCDCD CDCD 

CD CD CD CD CD 
CDCDCD CDCD 

cdodcd'cdcd 

CD CD CD CD CD 



NOTE: The ovals beneath the boxes were used by NAEP for 

machine scoring* You will not have to grid or darken 



in the ovals for your scoring. 
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CHAPJE& 10 
SCORING THE ASSESSMENT BOOKLETS 

e 

*i , , . 

Since the students record their answers directly in the booklets, 
the scoring task is primarily a prpcess of transcribing each stu- 
dent's responses to some type of coding form that can bp used to 
tabulate and analyze the results. Once this transcribing task is 
completed you can either hand tabulate or keypunch the data for 
computer processing. . 

All questions are in a multiple-choice format except question 12 

* 

on page 15. This open-ended question requires hand scoring using 
guides that are included in Appendix B. Before deciding to spend 
the time to hand score it, you may want to carefully review this 
question and Its scoring guide in terms of its importance to you. 
Whether or not you decide to score this question, we recommend that 
you do administer it in order to maintain continuity between the 
booklet and the cassette tape. Your decision to score or not to 
score need not disrupt the administration of the survey. 

.The amount df time and effort necessary to transcribe the students* 
answers from the bobfclets to the coding'forms will be directly re- 
lated to the number of students assessed. A conservative figure 
of 5 minutes per booklet could be used as a planning estimate. 
Thus, if you assessed 100 students, the clerical time involved to 
transcribe the" identification information on £he cover page of the 
booklets (i.D. 'number, grade* sex, birthdate and race) plus the 
responses to the survey questions, plus the questions on the back 



page would be 500 minutes or about 8 hours. (The estimate of 5 
minutes per booklet is quite conservative and you most likely will 
experience a faster transcribing time.) Illustrations of two saniple 
coding forms you might wish to use are shown below. Form A is' pro- 
bably the simplest format to use if you are interested only in 
looking at 'the group performance level of your stiJite^ on each 
survey question. Keep in mind that this format would not allow ;/ 
you to make subsequent data tabulations by sex, race, level of 
parents 1 education, etc., since the coding form does ndt identify 
these variables. 



FORM A CODING SHEET FOR* HAND TALLYING RESPONSES , 


Question 
Number 


Tally of Response Categories 


i 

Total 


% 


1 . .speech 
..religion 
..py^chase 
. .trial 

. . I don't know 


fW 

tw 1 

/HJttiirHJrHimjrHJtHtrHjmj 

rw tm ' " ^ — — _____ 
/// 


S 
(o 

si- 
ze 1 


6.7 
8.0 

J 3.3 


TOTALS 4 


IS 


IOOJ0 


2A..Yes" 

Mo 

/ 

. . I' don 't know 


S »_J . i KU 1 1 1 ft * 

/ 


GO 

/ 3 
X' 


irr o 

JL.7 


TOTALS ' 


75 


100.0 



30 



30 



If you plan to analyze your results by sex, race or any other 
variables, the Form B format (shown on page 33) would be -more 
appropriate. This format requires you to transcribe the student 
responses from the booklets to the coding sheets using a "code" 
system. Form B is designed so that the coded information can be 
readily keypunched for computer processing and analysis. The 
example below explains the coding system. ; . 

t • 

1. Which one of the following is NOT a constitutional right of 

American citizens? A 

O The right to freedom of speech 

<0 The right to practice the religion of one's choice 

The right to determine who can purchase and live 
in the house next door 

r- 

O The right to a fair trial when accused of a crime 
O I don ' t know. 

Assume that the student had selected the third response choice. The 
person doing the transcribing to Form B would code this response 
as' a "3". . If the student had selected the first response choice, 
code it as a "1*. If the student had selected the fourth response 
choice, code it as a "4". This type of coding scheme could be used 
to transcribe all of the student responses to Form B. If the stu- 
dent had selected the "I don't know" answer choice in the example 
above, you could code it as a "5^ since it is the fifth answer 
choice. For the sake of conformity," howeVer, it may be best to 
always code an "I don't know" response as a. "9" code. 



Shown below 1s a reduced copy of a booklet cover page and a student's 
response to questions 1 and 2. Form B, on the next page, shows how 
this information would appear in coded form. 



s , 





1. Which on* of the following U NOT t constitutional right of American 



cititens?' 



CD Tht right to freedom of speech 

CD The right to practice the religion of one's choice 

mm The right to determine who can purchase end live in the house next door 

CD *The right to * fair trial when accused of a crime 

CD I don't know. , 
2. A. Does the President ALWAYS have to obey the laws of the United 



mm Yes 



v > 



No 



I don't know. 



3^ - Dots a judge ALWAYS have to obey the laws of the United States? 



CD Yes 
cd No 

» 

mm I don't know. 

C. Does an urmy general ALWAYS have to obey the laws of the United 
States? 



Yes 

No. 

1 don't know. 
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. FORM B CODING SHEET FOR KEY PUNCHING RESPONSES 


Booklet Cover Information 


Survey Questions 




>- 


■'1 


I.O. No. 


r { V 
Grade 


(4) 
Sex 


. (5) 
Birthdate 


o (6) 
Race 


1 


2A 


28 


2C 


3A 


JO 


or 


ETC. 


00003 


0 8 




0 863 


A. 


3 


1 


r 














•i 


1 ^ 




0 










•» 






«« 



A listing of the questions and correct, answers for the 13- and 17- 
year-old assessment booklets is provided in Appendix C. Note that 
all questions asked of 13-year-olds were also asked of 17-year- 
olds. There are, however, a few questions in the 17-year-old book- 
let that do not appear in the 13-year-otfT booklet (questions 30B, 
31A & B, and 32A, B, C & D).» / 

You may o be able to complete your scoring by using present school 
personnel. If this presents a problem, however, do not overlook 
using volunteer help from outside theschool system. Interested 
parents could provide a very valuable service during your scoring 
task. • 
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CHAPTER 11 

• * ' . ANALYZING ANO COMPARING YOUR° RESULTS TO NAEP - 

Since" NAEP determines how well students in various groups achieve 
certain educational goals you will be able to compare your stu- 
dents as a group to these nati.onal groups if you fopow the re- 
quirements for replication. . —S; ■ 

/ ' " , <r 

It is very Important to keep in mind that 'NAEP will not be report- 
ing an overall score for the booklet. Each question stands alone 
and no attempt is made to report the percentage of students who 
answered correctly 50% or 75% or "90% of the questions. Students 
assessed by NAEP do not receive an individual score. Instead, for 
each survey item, the NAEP data answer the question, "What .per- 
centage of the nation's students know or can do this?" 

This chapter discusses the various data 7 results that will be re- 
ported by NAEP. It 1s important to note that the types of com- 
parisons you will be able to make will depend, upon the type of 
assessment you conduct (age-l*fvel versus grade-level). There are 
.basically two different types of results that NAEP will report: 

• question-by-question results 

• "theme" results (cluster of questions)^ 

question-By-Question Results - 

NAEP results will be provided by age-level and grade-level for 
each question. Thus, whether you conduct an age-level assess^ 
ment or a grade-level assessment, you will be able to compare the 
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results of your students .to the national sample. The 
percentage of students that respond correctly to a 
given question is called the p-value . The statisti- 
cal equation for determining p-values is: A 

let P * the percentage. of students answering a question 
correctly. 

\ ■* . '■ 

p m - J^c V. number of students responding correctly [ \ y lf|n 
* - n \ number of students responding I* 100 

For example, assume that 100 students respond to a question 
■ and that 86 select the correct' answer choice. The p-value 
for this question would be: 

86- 

P = TooT = -86 X 100 = 86.095, " 

Theme Resiil ts *' 
NAEP results also will~be provided by "themes". A theme 1s 
comprised of a cluster of questions that all measure a simi- 
lar knowledge, skill or attitude. The citizenship questions 
are categorized according to four general themes: 
• •Political Attitudes 

• Political Education^ 

• Political Knowledge ; 

• Social Attitudes ' 



NOTE: 



Theme results will be reported by .age-level only. 
Thus, if you conduct a 13- and/or 17-year-old assess- 
ment, you will be able to make theme comparisons. 



NAEP does not plan.-to compute and report grade-level 
- „(8th and/or 11th goalie) theme results at this time. 

The NAEP theme results will be reported as the mean (average) 

percentage of students that responded correctly to each theme. 

This statistic Js called the mean p-value . The statistical « 

equation for computing the mean p-value 1s: 

— • ••»•'. 

Let P * the p-value for a theme. 

— Pi + P 

P * 1 2... _ sum of p-values 
rtq ' number of questions 

For example, the "Social Attitudes" theme is comprised of 
test questions 3ABC* 6ABCDE,. 15ABC, 20 and 27.^ Assume the 
Individual p-values for each question were as shown below: 

Question p-Value(%) Question " p-Value(%) 



3A 


72.4 


15A •'. 


62.8 


3B 


93,6 


15B 


71.1 


3C 


. 64 . 9 


15C, 


89.9 


6A 


88.1 

V 

/ 


20 


90.7 


6B 


85.5 


27 


82.4 


6C 


' - 89.0 






6D 
6E 


91.7 
93.4 

• 






Then, p 


72.4 + 93.6 + 64.9 + 88.1 + 


. r. 82.4 




. 13 





/ 



p * 107 5., 5 3 82.7% 
13 

,.. 36 



Thus, the mean (average) percentage of students that re- 
spqnded correctly to the cluster of questions comprising 
the Social Attitudes theme would be 82.7%. 



A listing of all questions and their -categorization by ;; 
theme is provided in Appendrx C. Note that each question 
4 is also keyed to a citizenship- objective.. While NAEP 
does not plan to compute mean p-values for each objective, 
this information is provided should you want to analyze 
your data by objective. The citizenship objectives are 
shown in Appendix D. 

Additional NAEP Reporting Groups . 

In addition to reporting national results, NAEP also will 
report on five subgroup populations. Again, depending 
upon the type of assessment you conduct (age-level versus 
grade-level), you will be able to compare your results to 
some or all of these subgroups. The five NAEP reporting 
subgroups are shown below. w f , * 



■ V 



Geographic Regi on . . .. v ' Northeast , Southeast , 

Central , West 

Sex ..Male, Female V 

Race. 7 .White, Black 

// ■ 

Level of Parental Educati on... No Hfgh .School Some High 
, school, Gradyated High 

. J School , Post R|gh 'School 

Size and Type of Community. . ^ ..High-Socioeconomic (Affluen 

1 ' " . Urban; low-socioeconomic 
\ e i - (Disadvantaged) Urban; 

: ; Riiral; Urban Fringe; Main 
- or7 Big City; Medium City; 
d/^ri-T* .Small Places. 



Definitions for these five subgroup populations are shown 
1n Appendix E. 

j 

Summary of Comparison Options 

A summary of the various types of comparisons you will be 
able to make to the NAEP. data is shown below. A true NAEP 
replication (age-level ) does offer you more possible data 
comparisons than a grade-level assessment. However, even 
though you conduct a grade-level i assessment (8th l and/or 
lltH) and analyze your results for those students who are 
13- and/or 17-years-old, there are several comparisons you 
will be able to make. . 



Type of 
Assessment 


NAEP RESULTS 
Nat'l Reaion * Sex 


Race 


Parent ' s 
Educ. 


STOC* 


Age-Level 














A, Question- ^ 














By-Question 


Yes 


Yes 


Yes 


Yes 


Yes • 


Yes 


B. Thenies 


; Yes 


Yes 


Yes 


Yes. 


Yes 


. Yes 
















Grade-Level 

x A. Question- . 
8y-Que$t1ori 

B. Themes 

: 1 


* Yes 
No 


Yes 
No 


Yes 
No 


No ; 
No ^ 


No 
No 


. > 

,.:.'y ' '. * 

NO j . 
No 



/ 



' * Size and Type of Community 

NOTE: While NAEP does not plan to report grade-level results for all 
categories, you could/ compute, for example, the NAEP tneme re- 
sults if this would be important to your assessment comparisons. 
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Accuracy of Comparisons Between National Assessment Data^ 
And Your Data • 
The measurement of student attainments, like the measure- 
men t of the length of physical objects, can be done with 
reasonable accuracy if the Appropriate tools are used and 
the proper procedures are followed. However, even under 
the best of circumstances, ehrors may creep into our meas- 
urements that may cause us to wonder what^ponfidence we 
should place in the findings. Errors could result from 
happenstance, such as a firebell going off during the mid- 
dle of -the assessment session or they may stem from a lack 
of proper' planning, such as, scheduling the assessment at 
a time when honor students are out of school pursuing in- 
dependent study. Either instance would probably bias the 
results, but the amount "of influence they would have on 
the overall results cannot easily be gauged. The best we 
can do* in order tQ minimize these and similar influences 
is to adhere to good assessment practices- 

Assuming that you do not experience situations similar to 
those discussed above, there remains an additional source 

4 

of variation that you will want to :tak£ into account when 
comparing data from your students to data collected by 
National Cessment. These reside in the NAEP data and 
result from the fact that NAEP assesses a sample of 13- 
and 17-year-olds rather than assessing all students. 
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Fortunately, tfils . variation (sampling error), unlike some *ther 
types of errors, can be .estimated very accurately and in n4ep 
samples is found to be very sjnal i. Nonetheless, errors of 1 ) 
type may be a reason for concern, especially when very accurate 
comparisons are required. 

For this reasjp, National Assessment, in publishing its findings, 
provides estimates of sampling error along with data showing - 
percentages of students responding correctly. The' "unit of 
error" that NAEP reports is called a standard error and is 
expressed in percentage points. Fo/ most questions, one . • 
standard error is usually found to be between 0.5 and 1.0 per- - 
centage points. The p-values as reported by NAEP, plus,and 
minus one standard "error t provides an interval that.we are 68* 
confident .would 'contain the p-values obtained had all eligibles 
been assessed. Thus, if NAEP reports a p-value of 60% for a par- 
ticular question and a standard error of 1,0 percentage points, 
you can be .68% confident that-the true p-value lies somewhere be- 
tween 59% to 61%. 

/. • 

If a higher level of confidence, is desired, you can view the 
NAEP p-values in terms of two standard errors, whi$h will raise 
the confidence level from 68% to 95%. For example, assume the 
same p-value of 60% as d.iscussed above. Two standard errors 
would double the interval. Therefore, you would be 95% confi- 
dent that the true p-value lies somewhere between 58% to' 62%.° 
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If you assess aVL of your age-el igibles or grade-el igibles,- there 
will not be any sampling errors associated' with your student data. 
If you assess a sample of e-1 igibles,. however, your- results also 
will contain sampling errors and you will need to determine your 
p-valus ranges as discussed above-. Appendix A provides information 

that will help you determine your sampling errors and make compari- 

t> < 

sons to the NAEP data. » 

Comparing i our Results to NAEP Results 
If you conduct your assessment following the procedures for repli- 
cating the NAEP model, you can make comparisons of your students 
to the national sample of students to see if any differences* appear 
in performance levels. .The formula for determining these dif- 
ferences?, is: 

Let D = the difference between your results and the 
NAEP results. 

, P a the percentage of students answering a question 
correctl> (p-value). * 

Then 0 *« . P yours'~~ ^NAEP ' 

For example, assume that your p-value for a particular question 
is 7K0% and NAEP's fcvalue is 70.0%. Using the formula above, 

\ D = 75.0% - 70.0% =5.0% 

Thi/S, :yo\sr students 1 performance level would be .5.0% above the 
.national sample. This formula should be used when comparing , 
results for each survey question, themes, or any of the subgroup- 
.populations, such as sex, race,. region of the country, etc. 
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Once the differences in p-values have been computed, you will want 
to determine if they are statistically significant. In other 
words, are 'the differences "reliable" differences? Remember that- 
each NAEP p-value idoes contain a small amount of variation due to 
sampling. You will need to take this- Into account when looking 
at the differences in p-values. The example below, explains how 
to determine if the differences are statistically significant 
(reliable). 

• First, look at the standard error figure associated with 
the NAEP p-value. Assume the p-value is 70.02 and the 
standard error is 0.8 percentage points. As described 
earlier, this means that you can be 68% confident that the 
true p-value lies somewhere between 70.0% + one standard 

j 

error, or 69.2% to 70.8%. Since your p-value was 75.0%, 
you can be fairly certain (68% confident) that your stu- 
dents did perform better. 

•'"If you want to increase the confidence level to 95%, 
simply double the reported NAEP standard error 
(0.8 X 2 = 1.6). this estimates that the p-value 
now lies somewhere between ZO.0% + 1.6 percentage 
points, or 68.4% to 71.6%. Since your students' 
p-value was 75.0%, you can be very certain (95% 
confident) that your students did perform better. 

• While a difference of 1% or 2% in performance levels 
could very well be statistical "iy significant, these 
differences may not necessarily be "educationally". 
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significant. In other words, a difference of 151 on a 
particular question or theme or subgroup would cer- 
tainly have greater educational implications than 
would" a .2% difference. 
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CHAPTER 12 , 
CONCLUDING REMARKS 1 

We hope this manual will be helpful as you plan and ' imp! ement 
your assessment. It was difficult to define a set of procedures 
that would be basic to everyone but we hope this manual has 
something to offer to each .of you, " * 

Keep in mind that this citizenship assessment kit is a pilot 

r - 

effort to determine how useful materials like these are~toT your 
assessment programs! Therefore, we ask that yau note any 'prob- 
lems you encounter in using the materials. We plan to send a 
questionnaire to each of you in a few months to solicit your 
reactions and comments concerning this pilot effort. Your ' 
candid evaluation of the kit's strengths. and weaknesses will 
help us decide whether assessment kits should be developed in 
other learning areas. 



: APPENDIX A 

DISCUSSION ON SURVEY SAMPLING 

•• / . . 

While techniquesof probability sampling have not been utilized 
widely in education, interest in applying these procedures is 
growing because of a general need to economize in the gathering 
of test information and other kinds of educational data. The 
purpose of this discussion is to acquaint readers who are not'* 
familiar with sampling theory or techniques with some basic 
facts, limitations and applications of jsampling so that they 
can decide whether to use*these procedures for conducting an 
assessment with the citizenship assessment kit. 

The first part of the discussion presents an overview of sampling 
its benefits and limitations. This,, information should be helpful 
in guiding the reader's decision whether to take a sampling ap- 
proach rather than assess all students. Thekest of this dis- 
cussion. presents_step-by-step instructions for two simplified- 
sampling procedures.. Neither approach requires extensive cal- 
culations, since information such as appropriate sample sizes and 
estimates of standard- error is presented in tabular form. For 
readers with backgrounds in .sampl ing statistics who. wish ;o ex- 
plore more elaborate sampling alternatives than will be discussed 
here., -.we refer them to two references that will be useful. 1 ' 2 

*_ i ' 

■ 1 

Cochran, W.G., Sampling Techniques . (Second Edition). New York: 
Wiley, 1963 

2 ' • 7 • 

- Kish, L. Survey Sampling . New York: Wiley, 1965 
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Deciding Whether Sampling Can Be of Benefit to You - 
The techniques of sampling enable you to obtain reasonably accurate 
estimates of performance levels of groups of students without hay- 
ing to. test each and every student. In other words, by testing 
a sample of students one obtains information: that can be used to 
predict how the total population might have performed had every 
student of a particular age or grade taken the survey. Quite 
obviously the use of sampling to acquire this information means 
that the costs and efforts associated with test printing, scoring 
and analysis probably can be reduced. Moreover, the entire assess- 
ment operation may be more manageable, because of *f ewer students 
being; assessed. It ii possible also that fewer. test administrators 
will need to be trained and those that are involved can be offered 
more training and supervision to help assure a smooth; efficient 
assessment operation. 

Sampling is not, however, without limitations. The most important 
limitation is- that its use precludes your being able to identify 
performance levels 'of individual students who were not included 
in the samples Thus, if you feel that it is essential. to have 
information on an every-student basis you need read no further, 
for in this instance at least, sampling is not appropriate. 

' A second limitation stems from the fact that sampling is. most 
efficient,' from a cost-savings point of view, in schools or school 
systems that have large enrollments. As a general rule, the 
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larger the enrollment, the smaller the'sample can be as a per- 
centage of the totyl enrollment and still provide accurate infor- 
mation. With small enrollments it may be necessary to assess 
almost all students in a particular age or grade to assure ac- 
curate estimates. To decide whether your school or district (for 
sampling purposes) falls into either a "large enrollment cater 
gory" for which sampling can be efficient or into a "small enroll- 
ment category" for which sampling may offer too few advantages to 
be practical, you will need to relate your grade or age enroll- 
ment data to the information provided 1n Table 1. To use this 
table, match your enrollment figures for grade 8 or grade 11 to 
the number in the cplumn labeled "total number of eligibles" that 
approximates your enrollment,' For example, if you wish to assess 
8th graders and your 8th grade enrollment is 797 find the row for 
800 eligibles. As shown by the numbers to the right, the sample 
sires necessary to produce reliable data range from 740 students 
(very reliable) down to 270 students (less reliable). In this 
example, a sample size of 740 is estimated in Table 1 to result 
in a standard error of 0.5 percentage points. Thus, if the per- 
centage of students responding correctly on a given question is 
found to be 502, we are 95% confident that the percentage for 
ell eligibles would be 50% + 2 standard errors. This provides a 
confidence interval of 49% to 51%. In contrast, a sample size ' • 
of 270 is estimated to result. in a standard error of 2.5 per- 
centage points. With this size sample, a 952 confidence interval 
when p = 50% would be 45% to 55%. 
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TABLE 1 / 
TABLE OF APPROXIMATE SAMPLE SIZES 
FOR VARIOUS LEVELS OF SAMPLING ERROR 



Estimated Site of 
One Standard Error* 



tal Number 


0.5% 


i.d.% 


2.5% 


100 


100 


100 


100 


150 


150 


150 


110 


500 


200 ' 


185 


135* 


*5 C fi 


,250 


230 


160 - 




* 30.0 


270 


175 


350 

www 


340 


. 310 


190 


400 


390 


•/. 350 


200 


450 


430 


380 


215 


D0U 


475 


415 


225 




570 


485 


240 


700 

/WW 


6*50 • . 


550 


255 


o n f\ 

800 


740 


600 


27W 


900 




665 

WWW 


275 


'1 A A A * 

1000 


910 

< 


715 


. 285 ■- 


1250 


1110 


Q *3 C 

QO w 


300 

WWW 


1500 


1300 


940 


' 11$ 


1750 


1500 


1030 


325 


2000 


' 1667 


1110 . 


335 


2500 


2000 


1250 


350 


3000 


2300 


1365 


355 


4000 


2850 , . 


1540 


365 



*The percentage •{?) answering correctly + 2 times standard error 
provides an interval that we. are 95% confident would contain the 
percentage answering correctly of all age eligible^ If P - 5*% 
and one standard error equals 1*0%, the 95% confidence interval 

is 48% to 52%. ' M 
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The enrollment used in this example is such that sampling ought 
to be considered seriously as an option. Whether to assess 740 
students rather than a minimal sample of 270 should be decided 
after taking into account the sampling error that the different 
sample sizes are likely. to produce. Strictly speaking, this 
decision is .not one even a sampling expert should make. Rather 
it should be made by school personnel who are familiar with the 
intended uses of the assessment information and who have a feel 
for the level of accuracy needed.. 

The information in Table 1 also can be used to estimate sample 
sizes for age-populations. Just use 8th grade enrollments as 
an approximation of the number of 13-year-olds in your school 
and use 11th grade ^enrollments as an approximation of the number 
of 17-year-olds and 7 proceed as discussed previously. 

In, studying this table it becomes evident that sampling is poten- 
tially most useful for schools or districts in which grade enroll- 
ments exceed 400-500 students." With smaller enrollments it is 
necessary to test almost all students to obtain precise* estimates 
from- sampling. Even if greater error can be tolerated in the 
findings one must assess nearly half of the students. 

A finaT disadvantage of sampling to be discussed here stems from 
possible disnjptions to ongoing classroom instruction. The step- 
by-step procedures to be discussed later in this appendix require 
that students be'sampled randomly from a school-wide or district- 
wide listing by grade or age-criteria. Selection of students from 
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such a listing means that at the^ime of testing, the students 

; ■ $ • ■■ . , . .... • '• • 

will need to leave their regularly assigned classes and come to 

a central site to participate. It' iS! highly unlikely that random 
selection procedures would ever result in all' students within 
any particular classroom being selected. Moreover, it is impos- 
sible to predict in advance of the actual selection, for example, 
whether one, six, eight or more students will need to leave Mr, 
Ross's class in American Problems- 

* 1 » ■ 

While sampling techniques do exist which minimize these kinds of 

problems, they are somewhat cumbersome and require more advanced 

statistical procedures. Readers wishing to investigate these 

possibilities further are referred to discussions on* "cluster 

sampling" .in textbooks such as Cochran (1963) mentioned, earl ier. 

Overview of Steps in Sampling ^ 
Sampling requires that a few more planning steps be undertaken 
prior to testing than is the case when all students are assessed. 
Also, there are additional steps of a procedural nature that re- 
sult from use of sampling. Steps of both. kinds are outlined 
below, before discussing some of them in mc y * detail. 
Planning Steps Involve d ? 

1. Identifying the pdpulation(s) of ir>terist for purposes < 
of comparing .your data to NAEP data: 

(a) 13-year-olds only 

(b) 17-year-olds only 

(c) both 13-year-olds and 17-year-olls 

(d) 8th graders 

\ A 

(e) 11th graders 

(f) both 8th, graders *r>\ Ji->r arc 

: . .... • so . ' " " " ■ * 



2. Deciding whether there are any subgroups of the. above, 
'populations for which information \s needed, such as 
males vs females, black students^vs white students, stu- 
dents attending school A vs students attending School B, 
etc. , '• 

3. Deciding the amount of /error you are willing to tolerate 
in the findings (see Table 1 for amount of estimated 
error associated with data from different sample sizes). 

4. Deciding which gne of the two sampling approaches de- 
scribed later in this discussion will be more useful to you 

»- « 

a » 

Procedural Steps Involved • \ ' ' _ " 

1. Preparing a listing of eligible students (see defini- 
tions in Chapter 6). The list should be arranged 
alphabetically >by student's last name. Other student 
information such as sex, race , birthdate or school attend- 
ed may also be needed and will be discussed later. 

2. Identifying the required number, of students to be 
sampled. 

3. Selecting randomly the particular students who will 
comprise the sample. 

* - 

4. Notifying students and having them complete the sur- 
vey booklet. " / 

5. Analyzing the data and computing* percent correct for 
each question based on sample data. 



• ., -- • /■-•'.■ ■ •': 

6. Determining the reliability of the comparisons between 
your data and those provided by NAEP. 

^ . • 

Choosing Which ^Sampling Approach Will Be Most Useful For Your Purposes 

As mentioned earlier, this manual is intended to help small and 

medium size districts, who may not have measurement or statistical 

■ . / 

c 

specialists on staff to conduct an assessment following NAEP pro- 
cedures. For this reason we are recommending two sampling pro- 
cedures that are straightforward, and easy to implement. If done 
carefully each procedure will result in reliable information. The 
main difference between them lies ir^ their appropriateness for 
examining certain types of educational ^questions. 

The first type of sampling that will be discussed is calle/J 
simple random sampling. * It "is perhaps most appropriate when only 
district-wide or school-wide information is required and information 
on certain sub-groups is not essential. For example, if your main 
interest is to answer the question "How well .do students in my dis- 
trict or school compare as a total group to NAEP findings?" simple^ 
random sampling can provide the answer- 
On the other hand, if acquiring information on subgroups of stu- 
dents is highly desired or essential, then a procedure called 
stratified random sampling is recommended. For example, if your - 
main purpose is to compare the relative performances of students 
in school A versus school B, stratified random sampling should 
be followed. This procedure will help Issure that your sample , 

. V f - 
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will automatically include enough students from each school to 
make Valid comparisons possible. 

These two approaches will now be described in more detail. 
Simple Random Sampling „ • . 
The word "random" may mean different^things under different 
circumstances. As the word_45^used here, it means providing 
an equal opportunity for each eligible studerft to be included 
in the sample. Thus* it' is necessary to prepare a listing of 

namfe of all eligible student's before the selection can take place. 

* • • ... 

Prepare the Listing 

With Simple random sampling any ordering of studenjf names 
is satisfactory but alphabetical ordering is frequently 
used. This may be done by clerks or in some schools can 
be provided by computer. In some instances *it may be help- 
ful to no£e additional student or school information alongside 
names when preparing this listing. If, for example, a specific 
time and date for the assessment y has been set, it may be useful 
to list the class assignment of each student for that time 
and date so, if selected, the teacher can be notified. Or 
it may be easier tq do this after the selection has been 
made inasmuch as this information will be needed only for 
the students actually sampled. 

The next step is to number the list sequentially from 1 to N, 
where N is the total number of eligibles. Nopstudeht's name 
should appear more than onde, nor should any one number appear 
more then once. ' 53 



Determine Sample Size 

Table 1 .should be consulted, to determine the sample size 
needed, taking into account the amount of error that is m . 
likely to result. from .samples of different sizes. Keep in 
mind that the numbers shown are approximate sample sizes 
which can be adjusted somewhat if necessary. F$r instance, 
if your auditorium holds 30Q students and the proper sample 
size for your particular enrollment is 310, go ahead and 
decide to sample 300 only. * • 

\ Randomly Select Students 
\ ~ — • > 

Aneffective means of selecting students randomly i? to 

use a table of random numbers fotind in many statistics 

textbooks or in mathematics reference books. Beginning 

with the firsVrandom number listed and taking additional. ; 

numbers in sequence, select students from the listing sheet 

who have corresponding numbers. If you J^ncounter a random 

number larger thata any number' on your 'listing sheet, disregard 

*:• * 

it. If you encounter the same number twice, also disregard 
it. Continue this process until the appropriate number of 
students have been selected. It is- necessary to keep a tally 
of the number of students selected so that you know when the 
proper sample size has been reached. 

Students so selected constitute the sample to be assessed. 
Every effort should be, made to Assess these stents and 
onl*y these students. 
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Stratified Random Sampling 

Stratified random sampling provides two'major advantages over 
simple, random sampling. First, it can be used to help. assure 
that "certain subgroups will be adequately represented in the 
sample so that these subgroup comparisons can be made. 4 ' For 
instance, if it is essential to be able to compare students in 
school A with students in school B, stratified sampling should 
be employed. . 

A second advantage is that smaller standard error's are often 
obtained, or.smaller sampie sizes can be used (as compared to 
simple random sampling) if stratified sampling is employed. 
One can achieve these benefits fro; this .type of sample even 
though you do not intend to make comparisons among the strati- 
fication variables or subgroups. . - 

- Preparing the Listing % 

. The idea behind stratified sampling is to divide the 
list of eligi^les into relatively homogeneous parts or 
strata such as separate schools, separate grades within . 
schools or. scholastic grade or previous test results — 

. "A" students, "B" students, etc. Stratification according 
to performance variables such as previous test results or 
student grade averages serves to improve ^the accuracy of 
estimates provided through sampling. Stratification accord- 
ing t0 reporting variables stirh as school attended may. 



increase accuracy but is intended mainly to assure that 
you will be^ble to make comparisons between subgroups 
of students. 1 

Whatever variables are used, eligibles should be classi- 7 
fied separately at the time the student eligibility lii;ts 
are prepared. For example, if your intention is to compare 
students from school A with students from schcol B you-would 
*pF&p$ire two y separate lists. Number student name*3 sequentially 
beginning with'l on -the first list and continue the number - 
sequence onto the second list as necessary. 

Determine Sample Size 

Refer to Table 1, /to determine the total number of students 

to be sampled, again taking into account the size of the 

* ■ *■ 

standard error that is likely to result from samples of 
different sizes. If the number of eligibles on each lasting 
is different, as is likely to be the case, it will be 
, necessary to properly apportion the total sample size to 
each of the separate listings. This is accomplished by 
computing the sampling fraction, f, and using it to deter- 
mine the number of students to be sampled f rom* each listing. , 

f - 



N / 
where n ■ total sample sii*> needed 
N •- .total number of eligibles 
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For example, assume that the district-wide 8th grade enroll- 
ment is 1000. From Table 1 you decide to sample 715 (one 
standard error estimated to be IX). The fraction (f) then 
is given by 

f •« 715 
1000 

< . - ' 

« .715 

Further assume that you have two listings — one for school 
A with 400 students and another for school B with 600 stu- 
dents (400 + 600 = 1000). In this example you wpuld se- 
lect 286 students (.715 X 400 » 286) from school A and 
429 students (.715 X 600 * 429) from school B. 

At this point it is necessary to note a possible limitation 
of Table 1 in estimating sample sizes when stratified sampling 
is used. Since Table 1 was designed to estimate sample sizes 
for the total population its use will not always assure a 
sufficient sample size for certain subgroups. 

For example, assume that your district enrollment for grade ' 
8 again is 1000 and you again decide to sample 715. However, 
now assume that 800 attend school A and only 200 attend 
school B. 

After applying the sampling fraction ,(f) to the eligi- 
bility list for each school you find that 572 are to be 
sampled from school A and 143 from school B. The 
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problem that arises is that the disproportionate enroll- 
ments between the two schools resultvin only 143 students 
being assessed in school B. . Table 1 states that when , 
the number of eligibles is onty 200, it. is necessary to 
sample 185 to assure the stated size of sampling error. 
Under the circumstances you have two choices: 

A. Sample 572 from school A and 143 from school B 
with the recognition that data from school B 
probably will' contain larger sampling errors than, 
data from school A. 

B. Sample 572 from school A and test all students 
in school B, thereby eliminating sampling errors 
for school B students. 

/ '■ ■ 

Randomly Select Students 

The procedure for 'selecting students is the same as that 
described on page A-9 except that two or more lists are 
being considered. As the random numbers are identified from 
• the table, determine whether any of the student listings con- 
tain 7 a corresponding number— If so, that student is selected. 
Upon reaching the proper sample size-for any one list, 
continue this process for the remaining lists until the 
proper sample size has bgen reached for each list. 

Be sure that these students and only these students so 
selected are assessed.. 
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'Determining Population estimates from -Sample Data 

The intent of sampling, of course, is to estimate how well the 

total group has performed even though not all students were 

/• • • ■ 

tested. / 

• . ■ / : ' 

In simple random sampling and 1n stratified random sampling as 
described here, the sample data provides a direct estimate of the 
entire age or grade population. . Thus,for each question, one need 
only compute the percent correct from sample data to obtain an 
esti.nwtte of percent correct for the population. The first step 
is to' find the proportion (p} of students in the sample respond- 
ing correctly in which , 

p - n c J number of students responding, correctly A 
n \ number of students responding / 

The percent correct (?) Is then found by multiplying'? X 100 
to give P. This process .is repeated for each question. Since 
some students who were sampled may not take the survey, the number 
of students responding may not equal the sample size in all cases. 

In addition to computing percent correct, on total .group perform- 

/. 

ance for each question, you may wish to compute percent correct 
for certain subgroups such as males versus females. The formula 
r for computing percent correct for subgroups is the same as used, 
for the total group except -only subgroup members are coupted. 
Again the first step is to find the proportion (p) of sample 
' • students in the subgroup responding correctly in which 
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n c 3 number of students in suboroup'responding correctly 
n — - number of students in subgroup responding 



Thus, to compute proportion correct ftfr males and finales, 

D B n c » number of males responding correctly 
M m n number of males responding 



and 

n c a number of females responding correctly 
p f 3 n number of females responding 

The percent correct (P) for each group is then found by multiply- 
ing each p X 100. 

.0 

The reliability of subgroup data depends partly on the number of 
respondents within those subgroups that happen to 4feve been select- 
ed in the sample. Males and females, as a subgroup, probably can 
be broken out since approximately half- of the sample will be'com- 
prised of males and half of females. For other subgroups such 

* 

as white, or black students, it may be the case that too few whites 
or blacks were selected in the sample to result in reliable, esti- 
mates. Procedures for judging the reliability of the findings will 
be discussed next;. 

Estimating the Reliability of Your Findings 

-/■.*'.- . ' ' ■ ■ "* . 

After you have completed the computations, .question by question, 

for determining the percent correct for total group and for sub- 
groups of interest, you may also wish to determine how reliable 
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these data are. For instance, how sure can you be that the sam- 
ple data accurately estimates the findings you might obtain had 
all students been assessed? Or, how sure can you be* that subgroup 
data are really accurate estimates? * 

To assist in making these determinations we have provided esti- 
mates of standard error that usually resuft* from samples of 
various sizes in Tables 2-9. Error estimates in these tables tend 
to be larger than those you might obtain if you actually calculate 
the error in your particular sample according to standard statisti- 
cal formulas. In other words, you can be reasonably sure that 
standard errors that you might compute from your sample will not be 
larger than those estimated by the tables. 

In order to use the tables you will need to provide three kinds " 
of information: 

(1) your sample size . You must know the number of students 
in the total sample and the number of students in the 
sample making up particular subgroups such as the num- 
ber of males and the number of females. 

(2) percent correct . You must know for each question the 
percent correct for the total sample and the percent 
correct for any subgroups of interest within the sample. 

(3) sampling fraction . • You must compute the sampling fra'c- " 
tion (f) for the total, sample (see page A-l 2). Depend- 
ing upon the value of f you" will need to refer to dif- 
ferent tables*. The tables cover values of f from' 0.0 

to .79. . 61- 



An explanation of the proper use of Tables 2-9 wilT t^e provided by 
a^example. Suppose thaVthere are 1000 eligibles 'in your district 
and you decided from Table 1 to sample 715 students because that 
size sample was estimated (by Table 1) to produce sampling errors, 
of about 1%. Suppose further that you stratified by school accord- 
ing to the sampling fraction .715 so that 286 of 400 eligible stu- 
dents were sampled from school A and 429 of 600 eligible students- 
were sampled from school B. -After sampling, you counted the num-' 
, ber of males and fema\es in the sample and found 370 males and 
345 females. - ' 

For question one the oercent correct for -each qrouo'was found to 
be as follows: . ' " ■ 



P total - 


* 64% 


p 

school A 


■ 83% 


P sch6tfl B 


= SU 


p 

males 


= 57% 


p femaVes 


= 71% 



We 'will first determine the standard error for the total group. 
Since different tables -provide, data for different ranges of f, we 
go to Table 9 because this table is for f * .70 - .79 and f 1n 
this example is .715. Next we locate'the sample size that is 
closest to 715. This is found. to be 700: We ther^ locate the per- 
cent correct that is closest to 64%. At" the intersection of the * 
row showing sample size and the column showing percent correct is 
.a number, -.90, which is the standard error (<fe) given in percen- 
tage points. The percant correct, plus and minus two times this 
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standard error provides an Interval which we are 95% confident 
contains the true population value.' In this case this interval 
1s 62.2% - 65.8*. 

A second example will be provided- to show how the reliability of * 
male and female data can be determined. Entering Table 9 for 
a sample size of 370 (350) and for percent correct of 57% (55%) 
we find the standard error for males to be 1.33 percentage points. 

With a sample size of 345 (350) and percent correct of 71% (70) 

0 9 

Table 9 gives 1.23 as the standard error for females. ' 

The question of interest now is whether theVindings for males and 
females are Statistically significant. To test this we first de- 
termine the difference value (D) as follows: • 

* 57% - 71% • 
'! . « -14% . 

Next we compute the standard error of the differences as follows* 



se r 



se Pm* + se Pf 2 ' 



•y (1.33) 2 + (1.23 ) 2 



= 1.81 



* Many hand calculators compute square roots directly. 
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1s larger than plus or minus 2.0, then the difference (D) is signi 
ficant at the 95% level of confidence. In this example, since 

D -14% -7.73 . 

se D V 1.81 ~ . . , ' • ' 

the 5 difference between male and female findings is statistically 
significant. 

In a final example comparing school A and school B, we will show 

only the results so that you. can verify your own Computations. , 

■ . . ' .. "\ 

D * P A ~ P B 

Ir- ' 

se D '/"PA 2 +Sep B 2 ' 
* 1.37% 
= 20.38 

se 0 :■■ 

.. . - ' , . . '. \ 

Comparing Your Data to NAEP Data / '/*• 

VThe- number and type of comparisons' you will be able to make "J^V - "* 
tween your data and NAEP data dependon whether you assessed by 
age or by grade. More comparisons are possible if you have 
assessed by age as explained in Chapter 11 of this manual- ^ 

Another factor that possibly will limit the type and number of > 
comparisons i^j the number of students of a particular group that 
actually show^up in your sample,, NAEP gathers information nation 
wide in all types of schools, but it may be the case that your 



sample (or even your population) does not contain enough students 
of a particular type to make certain comparisons meaningful. For 
instance, you may have very few 'students whose parents have not 
completed high school. Your sample may not contain any students 
like this. Thus, it would be meaningless to compare your findings 
to the particular catego'ry of NAEP filings which Includes only 
students of this kind. 

Since the procedures for making'-comparisons are very similar to, 
the ones described previously in the section on Estimating the . 
Reliability of Your Findings, thpy/win only be summarized here. 

The main difference between wjiat follows and the procedures dis- 
cussed previously is that you must include NAEP data for percent^ 
correct and for standard errors along with your data 1n the for- 
mulas. • j 

■ > 

To compare your percent correct to NAEP percent correct for each 
question compute: t 

D = P yours - P NAEP ^ 



and se D * /se p 2 + se p 2 

u J "yours . NAEP 



and determine 7 if -JL. is larger than + 2.0. 

sen ~~ 
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to compare any subgroup such as males compute: 

. «taies « P your males , - P ^ a les 



and " * se n . « /se p . - + se P 

males V your piles 



2 

NAEP males 



(tales 



and determine 1f " males "is larger than +2.0. 



se 



Dmal 



es 



ESTIMATES OF STANDARD ERRORS FOR P-VALUES 
WHEN SAMPLING FRACTIONS ARE 



0 to .09 



If the sampte^slze n*lH and the percent correct 
(p-value) Is computed to be til A 

"s* •;• ■ To% ut0^zoi0/isi^i--zo%- ?1® . 

or or or , ttr \ or ; or or ' or ' nn* 

9SS: 90S Illll^flJllllll^ g§* 15*./ 55* ; 5 ° S 

one standard error equals: 



40% 45% 



2.13 
1.74 
1.51 
1.35 
1.23 
1.14 



2.94 
2.40 
2.07 

1 !'35 
1 ,69 
1.57 



1.0'^ 1.46 
1.00 1.38 



0.95 
0.^7 
0-.80 
0.75 
0.71* 
0.67 
0.55 
0.48 
0.42 
0.3§ 
0.36 
0.34 



1.31 

1.19 

1 . V 

1.03 

0.98 

0.93 

0.76 

0.55' 

0.53 

0.53 

0.49 

0.46 



3.50 
2.85 
2.47 
2.21, 
2.01 
1.86 
1.74 
1.64 
1.56" 
1.42 
" 1.32 
1.23 
1.16 
'1.10 
0.90 
0."73 
• 0.70 
0.64 
0.59 
0.55 



3.92 
3.19 
2.76 
2.47 
2.25 
2.09 
1.95 
.1.84 
1 i 75 
1.59 
1.47 
1.38 
1.30 
1.23" 
1.01 
0 .87 . 
0.78 
0.71 
0.66 
0.62 



4.24 

3.46 

2.99' 

2.67 

2-. 44 

2.2^ 

2.11 

M.99 
1.89 
1.72 
1.60 
1.49 
1.41 
1 .34 
1.09 4 

0.94 

0.84 . 

0.77 

0.71 

0.67 



4.49 4.67 

3.66 3.81 

3.17 3.30 

2.83 2.95 

2.58 2.69 

2.24 . 2.33 



2.11 

2.00 

1.82 

1.69 

1.58 

1.49 

1.41 

1.15 

1.00 

0.89 

0.82 



2*19 
2.08 
1.90' 
1 .J6 
1.64 

1.J3 
1,47 
1.^20 
1.04 
0.93 
0.85 



0.76 .0.79 
0.71 0.74 



4.80 
• 3.9/ 
3.38 
3.03 
2.76 
2.56 
2.39 
2.25 
2.14 
1.95 
1 .81 

1.69 
1.59 

1 *51 

1.23 

1.07 " 

0.96 

0.87 

0.81 

0.76 



4.87 
'3.97 
3.44 

i.07 

2. 30 1 

2.60 

2.43 

2.29 

2.17 

1.98. 

1.83, 

1.7.2- 

1 .52 

1.53 

1.25 

1.08 

0.97 

0.89 

0.82 

0.77 



4.90 
3.99 
3.45 
3.09 
■ 2.82 
2.61 

2.44 
2.30- 
2.18 
1.99 
1.84 
-1.7-2 
1.63 
K 5 4 
1.26 
1.09 
0.97, 
0-.89 
0.82 
0.77 
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ESTIMATES* OF STANDARD ERRORS FOR P-VALUES 
. WHEN SAMPLING FRACTIONS ARE 



.10 to .19 



If the sample size n= WMk and the percent correct 
(p-vaJlue) is computed to be Jllil 




one standard eVror/equal | : 



2.02 
1.65 
1.42 
1.27 
1.16 
'1.08 
1 .01 

0.95 

0.90 

0.82 

0.76 

0.71 

0.67 

0.64 

0.52 

0.45 

0.40 

0.37 

0.34 

0.32 



2.78 
2.27 
1.96 
1.75 
1.60 
1 .48 
1.38 
1.31 
1 .24 
•1,13 
1 .05 
0.98 
0.92 
0.88 
0.71 
0.62 
^0.55 
0.51 
0.47 
0.44 



3.31 
2.70 
2.33 
2.09 
1.90 
1 .76 
1 ; .65 
* 1 -55 
1.47 
1.35 
1.25 
1.16 
1.10 
1.04 
0.85 
0.74 
0.66 
^0.60 
.0.56 
0.52 



3.71 

3.02 

2,61 

2,34 

2.13 

1 .97 

1.85 

1.74 

1.'65 

1.51 

1.39 

1.30 

1.23 
1.17 
0.95 
.0.8.2 
0.74 
0.^7 
0.62 
0.58 



4.01 

3.27 

2.83 

2.53 

2.31 

2.14 

2.00 

1.88 
1.79 
1 .63 
1.51 
1.41 
' 1 .33 
1 .26 
1.03, 
0.89 
0.80 
0.73 
0.67 
0.63 



4.25 
3.46 
2.99 
2.68 
2.44 
2.26 
2.12 
1.99 
1.89 
1.73 
1.60 

1.49 
1.41 
^34 
1.09 
0.94 
0.85 
0.77 
0.71 
0.67 



4.42 

3.60 

3.12 

2.79 

2.54 

2,35 

2.20 

2.08 

1.97 

1.80 

1.66 " 

1.56 

1.47 

1.39 

1 .14 

0.98- 

0.88. 

0.80 

0.74 

0.70 



4.54 
3.70 
3.20 
2.86 
2.61 
2.42 
2.26 
2.13 
2.02 

t». Q 5 

1,7* 

1 .60 

1.51 

1.43 

1.17 

UQ+ 

0.90 

0.82 

p.76 

Q.71 



4.6* 

3.76 

3..25 

2.91 

2.65 

2.46 

2.30 

2.i6 

2.05 

1.87 

1.73 

1.62 

1.53 

1.45 , 

1.18 

0.92 
0.84 

0.78' 
p:73 



4.63 
3,78 
3.27 
2.92 
2.67 
2_a47^ 
2.31- 
2.18 
2.06 
1.88 
1.74 
1.63 
.1.54 
1 .46 
1 .19 
1.03 
0*92 
0.84 
0.78 
0.73 



TABLE 4 



ESTIMATES OF" STANDARD ERRORS FOR P- VALUES 
WHEN SAMPLING FRACTIONS ARE 



.20 to .29 



If the' sample size n=$S63ft and the percent correct 
(p-value) is computed to be Mi 




1.90 

1 .55 

1 .3.4 

1.20 

1.09 

1.01 

0.94 

0.89 

0.84 

0.77 

0.71 

0.67 

0.63 

0.60 

0.49 

0,42 

0.38 

0.34-, 

0.32 

0.30 



2.61 

2.13 

1.84 

1.65 

1.50 

1.39 

1.30 

1.23 

1.16 

1.06 

0.98 

0.92 

0.87 

0.82 

0.67 

0.58 

0.52 

0.47 

0.44 

0.41 



3.11 
2.53 
2.19 
1.96 
1 .79 
1.66 
1.55 
1 .46 
.. 1.38 
1.26' 
1.17 
1.09 
1.03 
0.98 
0.80 
0.69 
0.62 
-0.56 
Q.52 
0.49 



3.48 

2.84 

2.46 

2.20 

2.00 

1.85 

1.73 

1*63 

1.55 

1.42 

1.31 

1.23' 

1*16 

1.10 

0.89 

0.77 

'0.69 
0.63 
0.59 

0.55 



3.77 
.3.07 
2.66 
2.38' 
2.17 

2ior 

1 . 88 

1.77 : 

1.68 

1.53 

1.42 

1:33 

1.25 

1.19 

0.97 

0.84 

0.75 

0.68" 

0.63 

0.59 



3.99 

3.25 

2.81 

2.52 

2.30 

2.12 

1.99 

1.87 

1.78 

1.62 

1.50 

1.40 

1.32 

1.26 

1.03 

0.89 

0.79 

0.72 

0.67 

0.63 



4.15 
'3.38 

^2.93 
2.62 

2.39 
2.21 
2.07 
1.95 
..1.85 
1.69 
1.56 
•*1 .46 
1.38 
1.31 
1.07 
0.92 
0.83 
0.75 
0.70 
0,65 



4.26 
3.48 
3.01 
2.69 
2V45 
2.27 
2.12 
2.00 
1.90 

1.73 
'1.60 

1.50 
1.41 
1.34 
1.10 

0.95 
0.85 
0,77 . 
0.72 
0.67 



4.33 
3.53 
3.05 
2.73 
..2.49 
2.31 
2.16 
2.03 
1.93 
1.76 
1.63 
1.52 
1.44. 
1.36 
1.11 
0.96 
0.86 
0.79 
0,73 ' 
0.68 



4.35 
'3.55 
3.07 
2.74 
2.50 
2.32 
2.17 
2^04 
1.94 
1.77 
1.64 
1.53 
1.44 
1.37 
1.12 
0.97 
0.87 
0.79 
0.73 
0.68 
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TABLE 5 



ESTIMATES OF STANDARD ERRORS FOR P-VALUES 
WHEN SAMPLING FRACTIONS ARE 



.30 to .39 



If the sample size n= 



and the percent correct 



(p-value) is computed to be 




one standard error equals 



1.77 

1.44 

1.25, 

1.11, 

1 .02 

0.94 

0.88 

0.83 

0.79 

0.72 

0.66 

0.62 

0.59 

0.56 

0.45 
0.39 
0.35 
0.32 
0.30 
J. 28 



2.43 
1.98 

1.71 

1.53 

1.40 

1.29 

1.21 

1.14 

1.08 

0.99 

0.91 

0.86 

0.81 

0.77 

0.62 

0.54 

0.48 

0.44 

0.41 

0.38 



2.89 
2.36 
2.04 
1.82 
1.66 
1.54 
1.44 
1.36 
1.29 
1.18 
1.09 
1.02 
0.96 
0.91 
0.74 
0.64 
0.58 
CV53 
0.49 
0.46 



3.24 
2.64 
2.29 
2.04 
1.87 
1.73 
1.61 
1.52 
1.44 
'1.32 
1.22 
1.14 
1.08 
1 .02 
0.83 
0.72 
0.65 
'0.59 
0.55 
0.51 



3.51 
2.86 

2.47 

2,21 ' 

2.02 

1.87 

1.75 

1.65 

1.56 

1.43 

1.32 

1.24 

1.16 

1.10 

0.90 

0.78 

0.70 

0.64 

0.59 

0.55 



3.71 

3.03 

2.62 

2.34 

2.14 

1.98 

1.85 

1.74 

1.65 

1.51 

1.40 

1.31 

1.23 

1.17 

0.95 

0.83 

0.74 

0.67 

0.62 

0.58 



3.86 
3.15 
2.73 
2.44 
2.22 
2.06 
1.93 
r.81 
1.72 
1.57 
1.45 
1.36 
1.28 
1.£§ 
0.99 
0.86 
0.77 
0.70 
0.65 
0.61 



3.97 

3.24 

2.80 

2.50 

2.28 

2.11 

1.98 

1.86 

1.77 

1.61 

1.49 

1.40 

1.32 

1.25 

1.02 

0.88 

0.79 

0.72 

0.67 

0.62 



4.03 

3.29 

2.84= 

2.54 

2.32 

2.15 

2.01 

1 .89 
'1.80 

1.64 
1.52, 
1.42 
1.34 
1.27 
1.04 



4.05 
3.30 
2.86 
2.55 
2.33 
2.16' 
2.02* 
1.90 
1.80 
1.6^ 
1.52 
1.43 
1.34 
1.28 
1.04 
0.90 



0.90 
0.80^0. 81 
0.73 0.74 
0.68 0.68 
0.63 y0.64 
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TABLE 6 



ESTIMATES OF STANDARD ERRORS FOR P-VALUES 
' WHEN SAMPLING FRACTIONS ARE 



.40 ,to .49 



If the sample s1ze£n» jBBSftl and the percent correct 
(p-value) is computed to be. li|g§ 




one standard error equals 



1.62 

1.32 

1.15 

1.02 

0.93 

0*87 

0.81 

£•.76 

0.72 

0.66' 

0„61 

0.57 

0.5*< 

0.51 

0.42 

0.36 

0.32_ 

0.30 

0.27 

0.26 



2.24 

1.82, 

i.58 

1.41 

1.29 

1.19 

1.11 

i,05 

1.00 

0.91 

0.84 

0.79 

0.74, 

0.70 

0.57 

0.50 

0.45 * 

0.41 

0.38 

0.35 



2.66 

2.17 

1.88 

1.68 

1. v 53 

1.42 

1.33 

1.25 

1.19 

1.08 

1.00 

0.94 

0.88 

0.84 

0.68 

0.59 

0.53 

0,48 

0.45 
0.42 



2.98 

2.43 

2.fO 

1.88 

1.72 

1.59 

1 .49 

1.40 

1.33 

1.21 

1.12 

1.05 

0.-99 

0.94 

0.77 

0.66 

0.59 

0.54 

0.50 

0.47 



3.23 
2.63 
2.28 
2.04 
1.86 
1.72 
1.61 
1.52 
1.44 
1.31 
1.21 

•1.14 
1.07 
1.02 
0.83 
0.72 
0.64 
0.59 
0.54 
0.51 



3.42 
2.78 
2,41 
2.15 
1.97 

> 1.82 
1.70 

.1.60 
1.52 
1.39 
U29 

•1.20 
1.13 
1.08 
0.88 
0.76 
0.68 
0.62 
0.57 
0.54 



& 



3.56 

2.90 

2.51 

2.24 

2.05 

1.89 

1.77 

1.67. 

1.58 

H45 

1.34 

1.25 

1,18 

1.12 

0.91 

0.79 

0.71 

0.65 

0.60 

0.56 



3.65 

2.98 

2.58 

2.30 

2.10 

1.94 

1.82 

1.71 

1.63 

1.48 

1.37 

1.29 

1.21 

1.15 

.0.94 

0.81 

0.73 

0.66 

0.61 

0.57 . 



3.71 
3.02 

2.62 

2.34 

2.13 

1.97 

1.85 

1.74 

1.65 

1.51 

1.40 

1.31 

1.23 

1.17 

0.95 

0.83 

0.74 

0.67 

0.62 

6,58 



3.73 

3.04 

2.63 

2.35 

2.14 

1.-98 

1.86 

t-75 

1.66 

1.52" 

1.40 

1.31 

1.24 

1.17 

0.96 

0.83 

6.74 

0.68 

0.63 
0.59 
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TABLE 7 



ESTIMATES OF STANDARD ERRORS FOR P-VALUES 
% WHEN SAMPLING FRACTIONS ARE , 



.50 to .59 



If the sample size n= mms and the percent correct 
(p-value) >is computed to 

iilc - : liis 




one standard error equals 



I 



1.47. 
1.20 
1.04 
0.93 
0.85 
0.78/ 
0.73 
0.69 
0.65 
0.60 

0.55 
0.52 

0.49 
0.46 
0.38 
0.33 
0.29 
0.27 
0.25 
0.23 



2.02 
1.65 
1*43 
1.28 
.1.16 
1.08 
1.01 

0.95 
0.90 
0.82 
0.76 
0.71 
0.67 
0.64 
C.52 
0.45 
0.40 
0.37 
0.34 
,0,32 



2.41 

1.96 

1.70 

1.52 

1.39 

1.28 

1.20 

1.13 

1.07 

0.98 

0.91 

0.85 

0.80 

0.76 

0.62 

0.54 

0.48 

0.44 

0.40 

0.38 



2.70 
2.20 
1.90 
1.70 
1.55 
1.44 
1.34 
1.27 
1.20 
1.10 
1.01 
0.95 
0.89 
0.85 
0.69 
0.60 

0.54 
0.49 
0.45 
0.42 



2.92 

2.38 

2.06 

1.84 

1.68 

1.55 

1.45 

1.37\ 

1.3" ' 

1 . 19 

1.10 

1.03 

0.97 

.0.92 

0.75 

0.65 

0.58 

0.53 

0.49 

0.46 



3.09 
2.52 
2.18 

1.95 

1.78 

1.65 

1.54 

1.45 

1.38 

1.26 

1.16 

1.09 

1 .03 

0.97 

0.79 

0.69 

0.61 

0.56 

0.52 

0.49 



3.22 
2.62. 
2.27 
2.03 

1.85 
1.71 
1.60 

1.51 

1.43 

1.31 

1.21 

1.13 

1.07 

1.01 

0.83 

0.72 

0.64 

0.58 

0.54 
0.51 



3.30 

2,69 

2.33 

2.08 

1.90 

1.76 

1.65 

1.55 

1.47 

1.34 

1 .24 

1.16 

1.10 

1.04 

0.85 

0.74 

0.66 

0.60 

0.56 

0.52 



3.35 

2.73 

2.'37 

2.11 

1.93 

1*79 

1.67 

1.57 

1.49 

1.36 

1.26 

1.18 

1.11 

1.06 

0.86 

0.75 

0.67 

0.61 

0.56 

0.53 



3.37 

2.75 

2.38 

2.13 

1.94 

1.80 

1.68 

1.58 

1.50 

1.37 

1.27 

1.19 

1.12 

1.06 

0.87 

0.75 

0.67 

0.61 

0.57 
0.53 
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TABLE 8 

ESTIMATES OF STANDARD ERRORS FOR -P- VALUES 
. WHEN SAMPLING FMHONS ARE 



.60 tfi >0\ 
If the sample size n= SUS8 and th e percent correct 



(p-value) 1s computed to be^H, 




one standard error equals: 
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TABLE 9 



ESTIMATES OF STANDARD ERRORS FOR P-VALUES 
WHEN SAMPLING FRACTIONS ARE 



.70 to .79 



If the sample size n= 



/• 



and the percent correct 



(p-value) is computed to be ii^^< 




one standard error equals: 
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0.67 
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0.61 
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0.76 


0.80 
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. APPENDIX B '..']• 
. t SCORING GUIDES FOR QUESTION 12 

This open-ended question asked for. three responses concerning, 
"What could a person* 1n your community try to do 1f there were 
a local law which he Considered unjust? 16 The following scoring 
guide was- used by NAEP.; Should you decide to hand score this . 
question In order to make comparisons between your students and 
the national sample, you will need to follow the scoring guide. 

Note that three responses were solicited from each student. White 

some students will write three 'responses, others may only write 

one or two responses and some students will write none. The major 

scoring categories used by NAEP to categorize each response are 

printed In capital letters in the guide.. The scorers read each . 

student's response and assigned 1t to * spring category. They 

then recorded the three digit code ^/^r (f"icedin* the category) 

directly in the assessment booklet.'.. Ths-^rv've sets of avals at 

• . ■':■./'/■ 
the bottom of page 15 were used by f m Jjy? au could write the 

codes next to each response. . 

* - \ '4 

*NAEP .scored each student response as either acceptable, iift-.*ccGpt£bl 
no response &r "I don't know". Note that, the code numbers for all 
acceptable response categories begin with a ."1\ all u acceptable / 
response categories begin with a "2% an "I don't know'" response 
is codad "3S9 ' and a no response 1s coded "000". 

Eacfc student booklet should receive three sets; of code numbers. 
Assume that a student only gives one response. y Ti 2 scorer would 
categorize that response and then assign two sets -f "000" -codes 
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The "I dbn't know" code (399). should iws »3fl4 only if the student 
wrote this response in the booklet. . 

. ' \ ■ ...*• 

To receive an acceptable response to question 12, a student must 

mention at least two different ac cgg tablg scoring categories . For 
example, a studeht may give three reasons, but if "they all are 
scored as category 114 — LEGAL ACTION (COURT ACTION), SUE the stu- 
dent would not receive an overall acceptable score for the question. 
If a student mentioned two accept Able reasons (VU and 113) and one 
unacceptable 7 reason (251), _tS» «iu4ant ' wouW receive an acceptable 
score for the question since at letst tm Afferent acceptable 
reasons were given.. 

Beneath each major scoring category a?-a samples of actual student 

j 

responses that were scored duHns the NAEP 13- and 17-year-old 
assessment. Jhese sample r«?pwnses Should help to show how NAEP 
scored various student r«$£kff-aes. 

It is very important that all scorers be completely familiar with 
the scoring $Jide and that scores be assigned in a consistant 
manner* The following training procedure is suggested to obtain 

* 

consistent scoring among your scoring staff: 

• One person sftiiuld be designated as the "chief" scorer. This 

i ■ . 

person will be; responsible for training the other scorers . 

' I 

and supervising the scoring task. 

-P" ■ 



The chief scorer, 



and perhaps one other scorer, should 



select a sample of student responses. Reproduce (or type) 
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several copies of these responses. because they will be used 
to train other scorers later on. / 

• The two scorers should first review the scoring guide and - 
then bfi$;?r« tb score the sample responses. Any 'differences * 
in scores should be thoroughly discussed and reconciled. 

The .purpose of this pre-scoring session is to become very 
familiar with the scoring guide and to reach agreement on 
the scores assigned. 

• If other scorers are used, they should first review the 

scoring guide. Distribute the sets of sample student re- 

i 

sponses (without scores assigned) and let each "scorer work 
individually. After a third to a half of the sample, responses 
have bee/ scored by everyone, compare the scores to see if- 
any differences appear. (The chief scorer should refer to the 
paperi that were scored during the pre-scoring session to pro- 
vide scoring rationales.) 

i 

r 

• The chief scorer must serve as final arbitrator in settling . 
score differences. When the scoring becomes cpnsistent, you 
are ready to score the assessment booklets. 

• The chief scorer should constantly monitor and check to be 
sure that all scorers are scoring accurately and consistently. 
Conduct retraining sessions as necessary, using the sets of 
sample responses. 
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Pkg. 9, Ex. 12 ^ 

UNJUST L AW/C0MMUN ITY 



SCORING GUIDE 



Categories and sample responses are listed below. 



flfifi •= KO RESPONSE 

,110 ■ OTHER - RESPONSES THAT ARE ACCEPTABLE BUT DO NOT HAVE A 
ESTABLISHED CATEGORY AND ACCEPTABLE RESPONSES THAT ARE TC 
VAGUE TO CATEGORIZE 

Age 13 Try to stop that law and get rid of it. 

Try jbo have it changed. 

Ask the police about changing it. 

He could suggest a better law in its place. 

They could suggest changes in the law. 

Prove it unjust. 
Age 17 Complain. 

"Criticize. / 

111 = PETITION - CIRCULATE OR SIGN 

Age 13 He could write up a petition and have people in tt 
pommunity sign it then take it to court. 
Right "a petition. 

Take a peitibn around to withdraw the law. 
He could get a patishion signed by the community people. 
Age 17 If you didn't like the way things are get up a petition. 
Starting peition if necessary. 

h2 = circulate pamphlets, buhper stickers, billboards , posters 
~. buttons' 

Age 13 IJork fpr idea i.e., pass pamphlets.. 

Age 17 Work for idea ie pass pamphlets. 

I 

t 1 • 
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Pkg.. 9,-SBx. 12 ■ ' f 

• * 
Categories and sample responses are listed below, 

113 DEMONSTRATE, PROTEST, PICKET, CIVIL DISOBEDIENCE (AN ORGANIZED 

GROUP ACTIVITY OR SPECIFIES THEY 'WOULD DISOBEY THE RULE TO 
HAKE A POINT) 

Age 13 Have a strike against it. 

He could get a permit to picket. • * 

That person could protest. i 

Get a lot of people together and they would *block some 

official building so people couldn't get in our out 
He could make sigjis and protest it. 

Test case, bceak the law and bring it to court and try to 
prove that it is unconstitional. , 
Age 17 Demonstrate. 

As* a last' resort you could strike. . 

11H = LEGAL ACTION (COURT ACTION) , SUE 

Age 13 Go to court 

If he thought it was unconstititional he could take it to 
court. 

He could bring the laws to. the Supreme court of the US and 

let the nine cheif justice decide. 
The person could ask for ah appeal for the law. 
file a complaint. 
They could sue em. • 
Age 17 No samples. 

115 = USE THE HASS. HEDIA - APPEAR ON OR WRITE TO THE RADIO, THE 
TELEVISION ; WRITE TO THE NEWSPAPERS. _ 

Age 13 He could write a letter to the editor of his newspaper to 
protest it. 

Write to newspapers' to have everyone know arfauv; it. 
Write about it in the newspaper. 
Age 17 Use mass media. 

Write a letter to editor of newspapers. 
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Pkg. 9, Ex. 12 . 

••' '* . . ' i .'. " ■ • ' • 

Categories and sample^ responses are listed below. . 

* /■ 

1J6 «' USE/THREATEN TO USE THE ELECTION PROCESS - VOTE (INCLUDES 
CONDITIONAL "IF") , . RECALL, REFERENDUM, CAMPAIGN/. RUN FOR 
OFFICE 

Age 13 Try to have it voted on again. 

He could ask a friend to try to run for Mayor or something 

to try to change the law. . 
Not to vote for it next time, 
v Ask the mayor to have the city vote on it, &nd if it 
wasn*t passed it couldn't be enforced.^ 
He or she could demand to have a vote or a revote on a 
law. 

Age 17 i By voting r .making suggestions to authorities. 
'Being active in campaigns for local offices. 

117 = JOIN OR FORM A GROUP, LOBBY, JOIN A POLITICAL PARTY (DOES NOT 

INCLUDE SPECIFIC REFERENCES TO ELECTION ACTIVITIES) 

Age 13 Try to form a c >mmittee of people to help pass a new one. 
Age 17 Form a committee. 

Join the party and work. 

118 = ONL^Y GIVES A GENERAL MET HOI- OF INFLUENCING 

Age 13 He could speak out against* it. 

Report it to someone. * \ 

Send a letter about what he thinks is right. 

Go to the meeting and say he didn't like it. 
Age 17 Write letters. 

Speak out . _ 

Attend meetings. 
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pfcg. 9. S. 12 

■ * 

Categories and sample responses are listed below. 

— 4 

- WRITE TO, SPEAK TO, GET IN TOUCH WITH REPRESENTATIVE, SENATOR, 
OTHER ELECTED OFFICIAL; INCLUDES. GENERAL REFERENCE TO GETTING 
IN TOUCH WITH SOMEONE HIGHER OP, SOMEONE IN AUTHORITY (ASSUME 
* WRITE TO THEM' AND 1 TALK TO THEM* ' IN THIS -CATEGORY UNLESS 
FURTHER EXPLAINED) ' < - — ■ 

Age 13- Write his congressman and tell his feelings. 

He could talk to the people who passed it and tell them he 
doesn«t agree With it, ... 

^ Tell the city council i"f he had' other people agreeing with 

him. ,' / < 

He 'could go to the mayor and tell him what he thinks. 

He could write a letter or tell the people' to write letter 

to congess. » 
Go to the county seat. • 
Age 17 If you write to them 'enough. 

Talk to them. " • 

Write to them. 
Speak to them- 



120. = TALK TO, WRITE TO, GET IN TODCH WITH OTHER .PEOPLE, DISCUSS THE 
SITUATION, FIND. OUT WHAT OTHER PEOPLE THINK, BEET WITH OTHER 
PEOPLE ' ' •. r 

Age 13 Tell the people he doosn'-'t think' its fair . 
Get some friends to .help, him fight, it. 

Speak out and tell .other people about it and ask them to 
help. t 

Ask some opinions of other p.eople about what they think 
about it. • 

Make the neighborhood get together and try to see what 
they think about the sublect. 0 . 

Age 17 Talk to people (assume public) 
Speaking with .parents*. 

121 «. GENERAL REFERENCES TO GET-TING INFORMATION, BECOMING 
INFORMED/KNOWLEDGEABLE HEADING UP ON THS SITUATION. " 

Aqe 13 Look and find out how it could be made just. 
Age 17 No samples. 1 
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Pkq, 9, EX. 12 



Categories and sample Responses are listed belov. 

• / " c 

111 = INFLUENCE THROUGH TAKING ADVANTAGE OJ? POLITICAL FRIENDSHIPS 

Aqe 13 Hy Daddy\is .on the city council* I know the mayor and 
chief of police, 

Aqe 17 You see this very plainly in the precinct where the 
captain can help you in exchange for your* vote* 
My Daddy is on the <?ity council. I -know the ma^or and 
chief of police, \ \ 

121 = ATTEND CITY COUNCIL, COUNTY COMMISSIONER MEETINGS 

Aqe 13 He could come to a council meeting and protest, 

' They could bring up their passion in a city; council 
meeting. 

Age 17 When I grow up — If they have a meeting I could go and tell 
them, my opinion. 



250 * OTHER - VAGUE OR NONSENSICAL RESPONSES AND OTHER UNACCEPTABLES 
THAT DO NOT HAVE AN ESTABLISHED CATEGORY 

Age 13 Hove out of tiie State, 

He could raov® to. a different community; 
Better streets and alleys, - 
Better schools, 
• o More !a*rs. 

He coultf tear it up* 

Try to do something about it, * 
Aqe 17 No samples. 



V 



9 
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•pfcq. 9, Ex. 12 



Categories and sample responses are listed belc. • .» • 

251 '= BRIBE, OFFER MONEY X* 
Age 13. No samples./ 

Age 17 No samples. / ^ ■ * . 

212 ;= REFERENCES TO DISOBEYING THE LAW , RIOTING, USING flOLE^:* 

Age 13 Get a group of people and have a riot and beat some u ads 
* together. t 9 • x . 

He would protest in a violent way. 

Do not do what the law said. ' v » . 

Break, them. 

A^e 17 .No samples* / 

■ ■ . . '../.. 

253' * . OSE INTIMIDATION \ . ' / * 

* Age 13 No samples. 
Age 17 No samples. 

254 =. COHPLYING RATHER : *HAN CHANGING ; INCLUDES DOING NOTHING; OBEY 

. « . ; IT . ~~ „ .■» * 

' ' ' ■ 

Atje 13 They could lust let it go . * . 
Try to live with it. 
- Try to abide by the law the best that he knew how/ 
Try* to make the best of it. 
Or* he might not do anything about it. 
> Age 17 No samples'. * • 9 * / 

255" '= .NOTHING CAti BE DONE - 

; • •■ ■ . 0 * 

( Age 13\ Nothing £hat I know of. ■ •• . 

&ge 17. No samples. ~ 

399 = I DON*T KNOW 



. APPENDIX C 

LISTING OF ASSESSMENT QUESTIONS, 
CORRECT ANSWERS, THEMES AND OBJECTIVES 

'_.../ 

This appendix provides information that will, 'be important for 
your scoring and analysis tasks. Each column is explained below: 

■■' Columns A and B 

A sequential listing of question numbers for the 13- 
and J7-year-old booklets is shown in columns A and 
B. Note that most questions in the two booklets are 
identical. However, questions 30B, 31 A & B; and 32A, 

B, C & D in the 17-year-old booklet do not appear in 
the 13-year-old booklet. 

Column 

The correct (acceptable) answer for each question is 
shown in column C. ' For multiple-p*rt questions (A, B, 

C, etc.), NAEP will report the percent of students who 
, correctly answeVed each individual part. 

« Note that for some multiple-part questions, students have 
; to correctly answer two or. more of the parts in order to 
receive an overall acceptabl^score for the question. 
*• For example, look at question 2 which has parts A, B and 
. C. Only those students who correctly answer all three 
parts would receive an acceptable score for the entire 
quesjtion. (Questions 2, 4, 12- and 16 should be scored ac- 
cording to the notations in'column C). 
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For several questions, primarily the r attitudinal . . 
questions, the answer which is shown. may be viewed 
as "desirable" rather than correct or acceptable. 

For some questions, more tfjian one response choice may 
be scored as correct. In question 3 ,.for example, 
both the "yes" and the "I don't know" response choic 
are considered to be acceptable. 

Column D 7 * ' 

The theme membership for each question is shown in column 
D. If you plan' to analyze your resulTs*lby theme, you will 
need to compute a mean p-value for each of the four themes, 
(see Chapter 11), The four themes.are: Political Attitudes, 
Political Education, Political Knowledge and Social Attitudes 

Column E 

The citizenship objectives are coded in column E.' While 
NAEP will not be computing mean p-values for the objectives, 
this information -4s provided should you. want to do so/ Keep 
, in mind that the survey booklet you are using is only one 
of several booklets that was assessed by NAEP during 1975-76. 
Therefore, the survey coverage by themes or objectives should 
not be considered as. a full -scale citizenship assessment. 

Those questions rioted as "Background"; in column E were 
designed to gather curriculum related information in order 
. to address educators 1 concerns about whether students with 




different levels of exposure to or interest in 
citizenship issues would exhibit different levels 
of performance. NAEP plans to report student per- 
formance using some of the background ^questions as 
reporting variables. ' Since these reporting plans 

have not been finalized at this time, more infor- 

/' ' ' 

mation will be provided in the data reports. 
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LISTING OF ASSESSMENT OUESTIONS, 
CORRECT ANSWERS, THEMES AND OBJECTIVES 



' (A) 
13-Year-Old 
Question No. 


(B) 

17-Year-Old 
Question No. 


:c) 

Correct Answer 


(D)* 
Theme 


" (E) 
Objective 


1 

, a 


i 


The right to determine 

vfiiw uqii |juf uriudc anu 

live in the house next 
door 


Pol itical 

INIIUW 1 CULJC 

— 1 i 


II B 


2 A 


2 A 


Yes 


Political 
Knowledge 


» — - 
II A 


. 6 \' 
\ C . 

- 


B 


, Yes 


Political 
Knowledge 


II A 


C 

• 


Yes 

/ 

NOTE: For acceptable 
response to question 

Cf oLuusni iiiUdt cur**. 

rectly answer all 
three parts. 


Political 
Knowledge 

• 


II A 


3 A. 


3 A 


Yes; I don't know 


Social 
Attitudes 


I A ' ■•■ 

* 


B 


, B 


Yes; I don't know 


Social 
Attitudes 


t 

I A 


c 


. ■ c : 


Yes; I don't know 

NCTE: Both response 
choices scored as 
correct. 


Social 
Attitudes 


I A 

V 
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(A) 

13-Year-Gld 
Question No. 


(B) 

17-Year-Otd 
- Question No. 


" CO 

t 

Correct Answer 


CD) J 
Theme 


Objective 


-4 A 


•4 A 


Yes 


Political 
Attitudes 


IV C & D 


B 


B 


Yes 


Political 
Attitudes 


IV C Jk D 


C 


C 


Yes 


'olitical 
: H-*itudes 


IV C & D 


D 


. D 


Yes 


Police A 

At**' * « 


IV C & D 


E 


\ E 


v- Yes 


At-'.itufiTs 


i IV" C & D 


/ 


■ • 


NOTE: Ftv 2-eptable 
response to question _ 

4 - stud^; a t THi^t* rnir- 

• y wuvivv , -ULb o v wUI 

recti y tC 'rwiV any two 
of the five p«rts. 


• 


i 

! ■ 

i x 


5 A 


5 A 


No / 


Political 
Knowl edge 


. Ill D 


B 


B 


Yes 


Political • 
Knowledge 


III D 


6 A 


<• 

6 A i 

i 


: !o 


Social 
Attitudes' 


I E 


B 


B 


No 


Social 
Attitudes 


IE 


c 


C 

T 


Yes 


Social 
Attitudes 


I E 


' D 


Y 


No 


So-ial { 
Attitudes j 


I E 


E 


. E , 


No 


Social 
Attitudes 


E 

J . 



/ 
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(A) 

13- Year-Old 
Question No. 


- (b) 

17-Year-OH 
Question No. 


Correct Answer 


CD) 
Theme 


(E) 
Obj ective 


7 A 


7 A 

• i 


< 

No 


Political 
Know! edge 


III B 


B . 


B 


Yes 


Political 
Know! edge 


IIJ B 


C 


C 


Yes 


Political 
Knowledge 


III B 




8 A 


No • 

* 


Political 
Know! edge 


V A, 


B 


B 


No 


Political 
Knowl edge 


V A 


c 


C 


Yes 


Political 
Knowledge 


V A i 

/ 


9 A 


9 A 


Yes , • ■ . • ■ \ 




Political * 
Knowl edge 


III B 


B 


B 


No 




Political 
Xnowledge 


II B 


C 


C 


No | 


Political 
Knowledge 


II. B 


,10 A 


It. A- 


True, about me 


Political 
Attit. Jdes 


IV a 


B . 


B 


True abbot me 


Political 
Attitudes 


IV A 


C 


C 


True about me 

• 


Political 
Attitud.es 1 


,v A 


1 1 A 


11 A . 


No 


Political. 

/Know*- *k je 


UI B 

\ 


b- ; 


B 


Yes ,."••< 


Political 
Knowl edc-,0 

j 


III B '.. 


c 

V 


C 


Yes 


r 

Political 
Knowl edge 


Ill B • 


• . .,, 12 

• 


12 

* 


" m . See Appendix B for 
scoring keys and 
guides* 


Political ; 
Knowl edge 


' T T P 



(A) 

13-Year-01d 
Question No. 



(B) 

17- Year-Old 
Question No. 



Correct Answer 



"(D) 

/ 

Theme 



(E) 
Objective 



13 



14 A 
B 



15 A 



16 A 

B 
G 



. 13 



14 A 
B 



15 A 



16 A 

/ B 
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No, because he is 
protected by an 
amendment to the 
Constitution, 



It varies with the 
population of the 
state. 



Political •. 
Knowledge 



Political 
Know! edge 

Political 
Knowl edge 



I wouldn't be able to 
say one way or the 
other if the person 
cfould be trusted. 

I wouldn't be able to 
say one way or the 
other if the person 
could be trusted. 

I wouldn't be able to 
say one way or the 
other if the person 
could be trusted. 



Disagree 



Agree 



Disagree 



NOTE: . Question 16 . 
should be scored in 
two parts. For an 
acceptable response to 
the first part, stu- 
dent must correctly 
answer both Part A and 
B. For an acceptable 
response to the second 
part, student must cor- 
rectly answer Part C. 



Social 
Attitudes 



Social 
Attitudes 



Social 
Attitudes 



II B 



'olitical 
Attitudes 

'Qlitical 
Attitudes 

Political 
Attitudes 
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III B 
III B 



I A 



I A 



I A 



III F 
III F 
II B 



(A) 

13-Year-01d 
-Question No. 


(B) 

17-Year-01d 
'Question No. 


Correct Answer 


(D) 
Theme 


(El 
Objective 


17 


17 


To determine guilt or 
innocence 


Political 
Knowl edge 


t t n 

II B 


18 A 
B 


18 A 
B 


NOTE : Consul tants have 
advised NAEP not to 
report results. His- 
torians disagree on 
the validity of the 
question. 






19 A 


19 A 


Yes 


Political 
Knowledge 


II B 


B 


. *B 


Yes 


Political 
Knowledge 


II B 


c 


C 

/•'■' 


No. 


Political 
Knowledge 


II B 


D $ ' c 

■ / - ■ t 


' D ' c 


Yes 


Political 
Knowledge 


II B 


E 


E , 


No *' 


Political 
Knowledge 


II 3 £ 

V 


20 


20 


The family should be 
able to move in; skin 
color shouldn't make 
any difference. 


Social * 
Attitudes 


I E ; * 

V 


21 


21 


False 


Political 
Knowl edge" 


T T t r" 

III E 


.22 A 


22 A 


No 

* 


Political 
Attitudes 


II B . 


B 


. B 


No 


Political 
Attitudes 


II B 


c 






r u i i l i v* a i 

Attitudes 


II B 



91 



v . .... .. • 

.* 

» • 

erJc r -* 



. (A) 
,13-Year-Old 
Question No. 


17-Year-01d 
7 Question No. 


CO 

Correct Answer 

7 


(0) 
Theme 


(El. 
Objective 


23 A 


23 A 


No 


roi iticai 
Knowledge 


T t n 

II B 


. 8 


B 


No 


Political 
Attitudes 


II B_ 


24 A 


24 A 


No 


ro I i tica 1 
Knowl edge 


! V A 


B 


■ B 


No 


ronticai 
Knowl edge 


V A 


• C 


/ 

/ 


No 


Political 

Know! edge 

\ * 


V A 


25 A 


, 25 A 




ro n tica 1 
Knowl edge 


T T T n 

III B 


B 


' B 


Yes 

\ • •• 


Political 
Attitudes 


•II F 


& 


26 


Refuse to provide money 
for further military 
action 


Political 
Knowledge 


III B 


27 


27 


Try to stop the fight 
myself; Look for a 
ueacner or principal 
• .to stop the fight 


Social 
Attitudes 


IV A 


- 




NOTE: *Both response 
choices scored as 
correct. 


• 

✓ 


O .'/'■.. 


28 A 


28 -A . 

/ 


Nn 

liU 


hoi iticai 
Attitudes 


. Ill E 


B 


B 

/ 


No 


Political 
Knowl edge . 


II B 

I 
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/ T3-Year-01d 
/Question No. 


(B) - 
17iYear-01d 
Question No. 

i 


(.0 

'Correct Answer 


CP) 

Theme 


(E) 
Objective 


29 A 


29 A 


Almost every day; Once 
or twice a week; Three 
or four times a month 


Political 
Education 


Background 

t 


B 


B 


Almost every day; Once 
or twice a week; Three 
or four times a month 

NOTE: All three re- . 
§ponse choices scored 
as correct. 


Political 
Education 


Background 


30 

* 


30 A 


1 or 2; 3 or 4; 5 or 
more 


Pol itical 
Education 


Background 




B 

/ 


A good deal; Some 

NOTE: In Part A, all 
•three resppnse choices 
scored as correct. In 
Part B, both response 
choi eel scored as 

rnrrprt.- 


Political 
Education 


IV A 




31 A 


A good deal ; Some 


Political 
Education 


Background 




B 


A good deal ;. Some 

NOTE: Both resppnse 
choices scored as 
correct. 


Political 
Education 


* 

Background 




32 A 


True about the courses; 
Somewhat true about 
the courses 


Political 
Education 


Background 


* 


i. B ' 


True about the courses; 
Somewhat true about 
s the courses 


Political 
Education 


Background 


• •. \ 


c 


Not true about the 
courses 


Political 
Education 


Background 


• «* / 


D ••' 

' i. 


* 

True about the courses; 
Somewhat true about 
the courses 


Political 
Education 


Background 


.• "**V." ' . 

. 'T* ' ' . -.. * "■' .•' v ■ • 


■< ■ ■ 


NOTE:. Both response . 
choices -scored as 
correct for Parts A, 
B and £). 

J • ■ — 


» 


i 



(A) 

13-Year-Old 

• w ICUl wlU 

Question No. 


(B) _ 
Question No. 


(ci : 

Correct Answer ' 


(D) 
Theme \ 


(E) c 
Objective 


~T — ; — 1 r— 

. 31 A 


33 A 


Always; Often; 
Sometimes 


Political 
Education 


Background 


B 


B 


Always; Often; 
Sometimes 


Political 
Education 


Background x 


C 


C 


Always; Often; 
Sometimes 


Political 
Education 


Background 


D 


' D 


Always; Often; 
Sometimes 


Political 
Education 


-g^ckground 


E 


E 


Always; Often; 
Sometimes 

NOTE: All three re- 
sponse choices scored 
as correct. 


Political 
Education 


Background 
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APPENDIX D 
v . CITIZENSHIP OBJECTIVES 

The seven major citizenship objectives that were developed for 
NAEP's 1955-76 assessment program are shown below. Keep in mind 
that the assessment kit "booklet is only one of several booklets 
that was assessed by NAEP for the citizenship assessment. Since 
the questions in the survey booklet represent only a portion of 
the ^entire citizenship assessment, NAEP will not be reporting 
mean ,p-val ife's by objective. 

/ " ... ...... 

Subobjectives also are shown beneath each major objective area. 
A listing of the survey questions and their categorization by 
objective and qubobjective is .shown in Appendix C. Thi.s infor- 
mation is provided should you want to analyze your results for 
any of the citizenship objectives. . 

• < ; ' * 

I. SHOW CONCERN FOR THE WELL-BEING AND DIGNITY OF 
OTHERS . " 

A. Treat others with respect. 

B. Consider the consequences for others of their 
/ own actions. 

C. Guard saftiry and health of others./ 

D. Offer help to. others in need. 

E. Support equal opportunity in education, housing; 

employment, and recreation , 

F. Are loyal to country, to frifnde, and to other 
/groups whose values they alwre. 

G l Ar^ethical and dependable in work, school, and 
social situations. 
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II? SUPPORT JUST LAW AND THE RI&HTS OF ALL INDIVIDUALS. 

.. * ' * ' + 

A. Understand the need for*law... 

B. Recognize specif ic constitutional rights' and 

liberties . 

C. Defend rights andliberties of all kinds of \ 

people. 

D. Encourage^ethical and lawful behavior in others, 

E. Comply with public laws. 

*F. Oppose unjust rules, iaws, and authority by 
4 lawful means.- „ . 



III. KNOW THE MAIN STRUCTURE AND FUNCTIONS OF 7 THEIR 
GOVERNMENTS . , 

A. Recognize basic governmental purposes. 

B. Und^psliand the organization of federal and state 

government^. o % 
* C. Snow the political structure of their local 
• community. a i 

•D. Recognize the relationships of different levels 
of government. 

*■ ^ E. Recognize the* importance of politioal opposition 

and interest fcroups. . / « 

1 p F. Recognize that democracy depends on the alertness 
'/ and involvement of its citizens, and know how 

citizens can affect- government. , . « 

G. Know stricture 6f school and student government. 

. •; ! x ■ , 

IV. PARTICIPATE IN DEMOCRATIC CIVIC IMPROVEMENT. """ # 



» 



A. Believe that aach person's civic behavior is 

important, and convey this belief to others- 

B. Favor organized civic action where it is-needed. 

C. Actively work for civic improvement. v ' 

, D. Participate in local, sta_e, and national* govern- 
mental processes. 0 
E. + Apply democratic procedures effectively in small 
group 5 . 

*- y 

V. UNDERSTAND IMPORTANT* WORLD , NATIONAL AND LOCAL 
CIVIC -PROBLEMS. 

a! Understand" social conflict among individuals, 
groups, and nations and the difficulties in 
achieving peace and social harmony, ^ 

Bv Recognize how different civif policies may affect 
people's effprts to meet their economic needs. 
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C. Recognize, major environmental problems and are 
-aware~~of alternative civic solutions. 

D. See relations among civic problems and particular 

events. 

E. Generate gopd ideas about causes and solutions for 

civic problems. 



APPROACH CIVIC DECISIONS RATIONALLY. 

A. Seek relevant information "and alternative view- 

points on important civic decisions. 

B. . Evaluate civic communications and actions care- 

fully £s a basis for forming and changing their 
own views . * • . : „■ ■ 

C. Plan and organize civic ,tasks effectively." 

D. Support open, honest communication and universal 

education. 



HELP AND RESPECT THEIR OWN FAMILIES. 

A. Cooperate in home responsibilities and help pro- 

sj^vlde for other family members. 
Bi Instill civic' values and skills in other family 
members. " , 



i 
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' APPENDIX E 

DEFINITIONS OF NATIONAL ASSESSMENT REPORTING GROUPS 



Geographic Regions* 

NORTHEAST 

Connecticut 
Delaware 

District af Columbia 

Maine 

Maryland 

Massachusetts 

New Hampshire 

New Jersey 

New York 

Pennsylvania 

Rhode Island 

Vermont 



SOUTHEAST 

i 

Alabama 
Arkansas ■ 
Florida 
Georgia 
Kentucky 
Louisiana 
Mississippi 
North Carolina 
South Carolina 
Tennessee 
Virginia 
West Virginia 



CENTRAL 

Illinois 

Indiana 

Iowa 

Kansas 

Michigan 

Minnesota 

Missouri 

Nebraska 

North Dakota 

Ohio 

South Dakota 
Wisconsin 



WEST 

Alaska 

Arizona 

California 

Colorado 

Hawaii 

Idaho 

Montana 

Nevada 

New Mexico 

Oklahoma 

Oregon' 

Texas 

Utah 

Washington 
Wyoming , 



*Thc<se regional* subpopulation definitions are the same as those 
-used by the Office of Business Economics^- department of 
Commerce. 
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Results.are reported separately for males % and females. 



Race 

Results are reported for Whites" and piacks. While students were 
classified into otffer ethnic groups the actual sample sizes real- 
ized are insufficient for reporting purposes. • ' . 

Parental Education . ' » 

" . -/ * ■ 

Parenta^ education refers to the highest level of education re- 

• . ■ * 

ported by, the student for either parent. 

, " ✓ 

No High School — Neither parent has any formal education^. 
• . beyond the eighth grade," ~ 

. ..ii. 1 ' 
Some High Sch'ool — At least one parent has some formal 

education beyond the eighth grade, but 
-neither parent has graduated from high 

schoe* . " . 

Graduated From * 

High School '— At least etfte pare*nt has graduated from 
'. high school, but neither, parent has any 
formal education beyond high school. 

Post High School — At .least* one. parent has .some formal educa 
; *■ tion beyond high school including any 

, .business, professional or trade school 
training as well as college or university 
training. 



Size and Type of Community . • 

Community types are identified both by the,size of the community, 
and by the type of employment of the majority, of people in the . 
community. 

\ High-socioeconomic (A ffluent) Urban . Areas in or around cities 
. with a population greater than 200,000 where a high^ropo'rtion of 
the residents are in professional or managerial positions. 

+ * — 

Low-socioe conomic (Disadvantaged) Urban . Areas in or around 
cities with -a population greater than 200,000 where a high 
proportion of the residents are on welfare or are not reg- i 

... ' . \-.r 

ularly employed. ' • \'-.'« 

> 11. 
Rural_. Areas with a population under 1C,000 where most of 

the residents are farmers or farm workers. 

Urban Fringe. Communities within the metropolitan area of a 
city with a population- greater then 200,000, outside city 
limits and not in the High- or low-socioecOnomic u'-ban" groups. 

- • . . ■ 1 "*■ 

Main'Big Citv. Communities within the city limits of a city 
with a population over 200,0.00 and not included in the high- 
or low-socioeconomic urban groups. 1 - 

Medium City. 'Cities with populations between 25,000 and" 
200,000. * ~'~ 

Small Places/ Communities with a Q population of less than 25,000 

■ I. ' 

and not in the rural group « 
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